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Preface 



This symposium provides an opportunity to present and discuss Alberta's Educational 
Quality Indicators (EQI) initiative. At the 1989 meeting of the Canadian Educational 
Researchers' Association, McEwen and Zatko presented the conceptual framework 
and implementation plan for this initiative. This symposium brings together five of the 
project teams to provide their perspective on the initiative and to rcpon Lheir progress 
to date in developing local indicator systems which have the involvement and support 
of their respective communities. 

Alberta Education is collaborating with twelve school jurisdictions to develop indicator 
systems to measure the success of the educational enterprise in the province. The 
indicators will provide infomiation to assist in assessing the quality of educational 
programs and the delivery system by focusing on student outcomes. The proposed 
system has taken into consideration and retlects government policy and the goals of 
schooling. It addresses two essential questions: 

1 . Are smdents learning to their potential? 

2. Is the educational system supporting student learning 
efficientiy and effectively? 

The EQI initiative will focus on developing indicator systems, establishing 
procaiurcs, and reporting and disseminating the information to educational 
constimencies in Alberta. Figure 1 presenu^ the expected outcomes for these 
three components. 
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Report and Dissemination 



Figure 1 : Expected Outcomes of the EQI Initiative 
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The fundamental principle of the EQI initiative is that no single indicator, or even 
group of iniiicators, can fully describe the complexity of education. The proposed 
syitem wiLl include many indicators, measured by both quantitative and qualitative 
methods, for selected dimensions. EQI intends to include indicators organized in 
logical clusters, measured in different ways, using information firm multiple sources 
to describe educanon m such a fashion that meaningful interpretation and judgements 
can be nmde. The indicator system intends to enhance information about education for 
improved action in planning, policy and decision making. 

A four-dimensional model of education was developed to guide the direction of this 
imnanve. It consists of parmcrs (schooling, family and society), conditions (context, 
inputs and processes), student outcomes (cognitive, affective and behavioral) and time 
(grades 3, 6, 9 and 12) (McEwen and Zatko, 1989). 

The EQI initiative is sponsoring ten concurrent action research projects. One or more 
school junsdiction(s) within the zone of each Regional Office of Education was 
idennfied and invited to participate. Each school jurisdiction prepared a proposal 
based o:. the Terms of Reference and submitted it for approval The Planning and 
Policy Secretariat is providing funds to assist these jurisdictions to improve their 
assessment procedures and to share their results with others in the province. The 
mformaiion generated from Oie ten projects will assist Alberta Education to develop 
provincial mdicators. The interpretation and recommended directions of the local 
mdicator projects, together with other provincial initiatives, will provide a solid 
foundanon for the implementation of an efficient and effective information system 
which measures the success of the educational enterprise in the province. 

Each participating school jurisdiction is developing a locsl indicator system which 
includes the components identified in Figure 1: a set of indicators (including an 
interpretative firework, snident outcomes and points of reference), methods 
(to collect, analyze and interpret the information), and a reporting and dissemination 
strategy (to inform diverse audiences of the results). Evei>' project has three phases, 
each of approximately one year's duration. The first year, 1989/90, is developmental 
and the subsequent two years wiU result in field testing the protot>'pe sets of indicators 
and methodological procedures and then refining them. 

The five papo-s present different aspects of the principles behind EQI. The first three 
papers desCTibe the involvement of the community in determining the goals and 
pnonnes of education in their respective school jurisdictions. The last two paper: 
dMcnbe the development of a broader range of smdent outcomes to include the 
affecave and social domains. The five papers provide a good representation of the 
tjT^s of acnvines taking place during this developmental yeai- of the EQI projects. 



Nelly McEwen, Coordinator 
Educational Quality Indicators 
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The Quality Indicators Strategy: Invohing the Community 



Grande Prairie Public School Distria No. 2357 operates ten schools in the 
City of Grande Prairie and has ?» current enrolment of 4500 students. Five of the 
present schools arc elementary schools, one is an elementary-junior high school, 
one is a junior high school, one is a composite senior high school and one is an 
elementary-junior high school which offers programs and services to 150 multi- 
handic^pcd children and 350 children in regular school classes. A court school is 
also operated by the District under a contract with the attorney-general. 

Extensive Special Education services are offered by the district, including 
programs for trainable mentally handicapped, educable mentally handicapped, and 
aU levels of learning disabilities. Programs for gifted and talented smdents are also 
offered at elementary, junior and senior high schools. 

In July of 1989 the District entered into a contract with the Minister of 
Educauon of Alberta. The District agreed to conduct a resea'rh project for the 
Minister to identify indicator; tiiat could be used to deicraune the effectiveness of a 
school system in developing positive smdent outcomes in the cognirive, affective 
and behavioural domains. 

The Grande Prairie Quality Indicators Strategy has as its main emphasis and 
focus the stakeholders of education in tiie city of Grande Prairie. These 
stakeholders are defined as any person or group who: receives tiie educational 
product through instraction; uses the facilities or resources of the district; works for 
the district; contributes tiirough taxes to tiie operation of tiie district; has siblings 
attending school in the district; or hires or further educates tiie students of the 
distncL The project team has identified five major stakeholder groups including 
educators, parents, administrators, smdents and tiic public and has developed a 
strategy to gather tiieir opinions on tiie quality of education in tiie city of Grande 
ftaine. The targeted stakeholders identified quality indicators for measuring the 
effectiveness of tiie school district and its schools. This paper will demonstrate that 
while some of tiie initial results were very predictable, some of die indicators were 
unique to tiie Grande Prairie Public School District 

Introduction 

The Grande Prairie Public School District became involved with the 
Educational Quality Indicators smdy in order to find a way to measure tiie quality of 
education and tiie effectiveness of tiie product offered to students in tiie District 
The related literature on effective schools, high performance schools, quality 
educanon, and accountabiUty, provided tiie research team witii many questions 
regarding reportmg of school effectiveness. The project team used tiiree simple 
questions to guide its research in tiie area: 

Who wants to know? 

What do tiiey want to know? 

How do we report what tiiey want to know? 

The problem was not only to identify quality indicators but also to develop a 
method to measure tiie attainment of the indicators and, finally, to report tiie results 
so tiiat ail stakeholders could understand and use the information. Particular 
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JS.r^'^.?'r°'^'?f^^ what is feasible for 

Alberta practice. The "feeling" of the city was taken from stakeholders who have a 
pnmaiy interest for dctenmning the e£fcctivenr.ss of schools. An eMomoiy ™ 

?HfSlT£j^n!°*P^?y^. i" discussing the functions of explorato^ 
SeUnz, Jahoda, Deutsch, and Cook ( 1959) reported: 

Many exploratory studies have the puxpose of fonnulating a problem 
.JJ^J^^^I investigation or of developing hypotheses. An 
exploratory study may. however, h& . other functions: increasing 

wS'^S J^'""^ "^'^^ phenomenon he wishes to 
mvestigate m a subsequent, more highly structured, study, clariftdne 
concepts; es^blishing priorities for forther^earch; gS| 
SSlSJSi^"!^.?'^ possibilities for carrying out reWh in 
real-life settmgs; providing a census of problems regarded as urgent 
by people woriang m a given field of social relations. (p.51) 

The above autiiors suggest using tiie foUowing metiiods in this tVDc of rewarrh- a 

people who have had practical experience with tiie problem to be studied. 

f^.tnrc.'!?!^^ ^ ^'iP®^^ ""^^h °J«lcl was determined by tiie foUowine 
factors; tiie format would not allow any direct control of independent vSiable3 

by DonM^ tl^ " ^' l™j«=t was offered 

We contend that any dissatisfaction encountered among clie-ts 
deprived" of a new program is a drop compared wirtlie flSod of 
djs^nsfactton from taxpayers who di£overS,at milUon7Sa^e» 
spent 01^ propams that lacked a well planned method for 

0?~3'l7" *° aLmpUshS^iS 
SS1va«8?S~ » C.J^Ti^ of' 

g|«|Sorr5^s"~Thir^^^^^^^ 

Baker (1988) goes on to say: 

Achievement testing will not go away, and for good reason 
Students and, by impUcation, tiie'schools to which tiily^ S be 
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The team was anned mth many opinions from educational researchers such as 
Edmonds (1978), Brookover, and Lezotte (1977) on measuring school 
ettecQveness. The related literature on effective schools, high petformance schools 
qualiry education, and accountability, provide researchers witii many questions on 
the reportmg of school effectiveness (Frederick, 1987). The development of die 
strategy for die Grande Prairie Public School Distria smdy involved looking at 
vanous projects conducted in die United States (Codianni, & Wilbur, 1983) This 
research provided a solid basis for understanding die work diat had been done in 
the area, of school effectiveness and quaUty indicators of that effectiveness at die 
tune of the study. The team had to establish an appropriate mediod for measuring 
die quality mdicators of die Grande Prairie Public School Distria as defined by die 
targeted stakeholders. There were many established instruments and mediods which 
had already been employed by educational researchers in dieir attempts to define 
cntcna fw measurmg educational quaUty. The 'Summsry of Quality Indicators' 
smdy conducted by die Austin Independent School District cites indicators such as 
student achievement, college bound students, smdent diversity, basic skills 
attendance and drop out rates in comparing die effectiveness of dieir district widi 
odier distncts (District, 1987). The review of uie research on quaUty Indicators gave 
die project team a comfortable understanding of what to look for and provided a 
strategy to obtain the necessary data. 

Phase One: District Report Card 

Procedures 

As indicated earlier, the first step in phase one was to identify the 
stakeholders of education in die Grande Prairie Public School District The term 
stakeholder also had to be identified in terms diat would enable die team to target 
appropriate groups for input The second step was to establish a mediod or strategy 
to Identify what the stakeholders perceived die quality indicators of die schcxjl 
distnct to be. In order to attain quaUty information, die stakeholders needed to be 
mfomied of the overall objectives of die Grande Prairie Project The ultimate 
purpose of die study was determined and die mediodology was also established by 
Jie project team This information was presented to each stakeholder group as an 
inopduction to the sessions which were designed to obtain dieir opinions on quality 
inchcatore of education for die district After die introduction of die study, the 
s^eholders were asked to break into groups to brainstorm dieir quaUty indicators 
of an effective school district After Usting die items via brainstoiming, die groups 
were asked to rate dieu- indicators from most important to least importaiit Once the 
groups Usted then- Items in order of priority, die groups shared dieir results with 
°,"t^°!5^' ^« "^i«P of phase one waj *o analyze die data obtained from die 
stakeho der sessions The data was placed into a data base which identified the 
stakeholder group, die subgroup, die number of people in die subgroup, die date 
and die pnontized quality indicators of that subgroup. All identified quality 
indicators were dien placed on a spreadsheet and assigned a value of five points for 
a number one rating four points for second place, dirce points for diird place, two 
^^l^^ ^P^^°y^^ place and one point for fifdi place. The spreadsheet data is 
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Table 1 

Number of Stakeholders Involved in the Grande Prairie 
Educational Quality Indicators Strategy 



Groups 


Number 


Parents 


35 


Public 


36 


Students 


241 


Instructionai Staff 


229 


Non-Instructional Staff 


84 


Administrators 


36 


Total 


661 



The project team wiU report back to all stakeholders surveyed during phase one to 
solidify the cntena identified during the stakeholder sessions: After the 
Identification of stakeholders, the project team recruited volunteers to represent 
vanous community groups on a Quality Indicators Steering Committee. The team 
meets with this committee every three mondis to update them on the progress and to 
obtam feedback and duection on the project course. 

Preliminary Results 

The team is currcndy analyzing the data and deciding how best to rcpon the 
nnomgs. iTiere have been some interesting resuhs from die first phase of the 
surveys. All groups thus far have identified smdent achievement as one of the 
indicators of an effective school district There are varying opinions, however, on 
the meamng of student achievement The sessions conducted with the public sector 
indicates that items such as cmployability of graduates, respect, work ethic, 
amtudes and se'f esteem are considered to be measures of student achievement 
i'arent groups, Jucators and students hold standardized test scores, and district 
comparisons on provincial tests as measures of student achievement Research in 
/ ^^^^ indicates administtative leadership as a strong force in the measure 
of effecdveness. The uiitial results of the Grande Prairie survey indicate that few 
smkeholder groups rate Leadership in the top five indicators. Some groups, such as 
educators do not mention leadership at aU. The end goal of phase one of the study 

'^dZ^r^JZ\'!^^°?^'i^^^' "^^^^ ^" ^fl"^^ q"^^ indicators 

nuhH h^, P^^' d^"' s^^hool district has 

^mindy i^s^ °" performance based on die measurable criteria they 
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Reporting pcrfonnancc in this context was vciy well received by the school 
ooard and their enthusiasm provided much encouragement to the project team that 
they have chosen realistic and valuable objectives for the projccL The purposes for 
reporting on the quality indicators of an effective school district are two-fold. 
Firstly, It provides all stakeholders in education in Gr mde Prairie with meaningful 
feedback on the educational process and its results. This feedback will repon on 
cntena that is important and has been identified by the people who have a stake in 
what happens in tiie district It will serve as a common basis for discussions and 
quesnons on die districts' performance. The process of developing and reporting on 
these quality indicators wiU result in more knowledgeable stakeholders. More 
knowledgeable stakeholders equate to more knowledgeable questions and solutions 
to concerns. The feedback will also help foster a community or team approach to 
education in die Grande Prairie PubUc School districts. Stakeholders can truly 
become partners in the business of educating youth. 

Secondly, Uie disffict report card will serve as a blueprint for strategic 
planning to improve die delivery of education in CSrande Prairie. All participants in 
me study understand diere is nothing diat can be done about past performance. The 
district report card will, however, provide us widi a basis to target areas for 
improvement and a means of measuring whether or not the planned strateev is 
working. 

It is very clear diat Uie process of developing die report card will be just as 
valuable as the planned outcome. The educational industry can no longer make 
decisions and function in isolation from their stakeholders. Vicki Bowers (1990) 
makes a strong argument for more communication and feedback to stakeholders, 

TTie tragic thing is diat schools teach exciting, useful and important 
dungs to youngsters. By not openly imparting diat information to 
parents, schools deny die people who would be dieir best supporters 
the chance to speak knowledgeably. And that's self defeating, 
because well informed parents can counter the radical fringe more 
effectively than can school public relations people. (p.40) 

Phase Two: School Report Cards 

The second phase of uie study will involve d-.e development of school repon 
cards. Schools wiU be asked to volunteer to work on die development of individual 
report card which will reflect die opinions of their school's stakeholders The 
process of gadiermg diese opinions will emulate die mediods used in developing - 
distnct report card. The project team will work with die individual schools to 
idennfy and plan a time line to reach dieir stakeholders. It is expected that the 
quahty mdicators identified by die individual schools' stakeholders will vary. 

^" presenting phase one of the study to various stakeholder groups 
differences were wimessed in quality indicators as the smdy was presented to 
vanous school communities. An example of dus difference was in die results from 
educators in two separate elementary schools. Elementary school A listed indicators 
such as program equity, special programs, student motivation, and teachers 
mvolvement m decisions while Elementary school B listed different items such as; 
mamstrcammg, resources, inservice, and Board/Teacher relations 
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„c.^ «c K ^- ^ datJt bank from phase one of the study that can be 

used as a basis for discussion among the school stakeholder groups. For example 
Si n/r'^ °^ st^eholders. the presentations tS existing S 

dSi^'^T? crmmittees. Each committee represented a specific schoSl in the 
d^sOTO. Simihriy, the mstructional and non-instructional staff were contacted on a 
school by school basis. The project team wUl contact as many stakeaolders i 
Ewn.T^ group technique described in phase one. Further data 

^^^^^yhave to be accomplished using questionnaires or some son of mass 

romm J"^ resulting school report card from phase two of the smdy will include 
fSfo/c^Sl?'^??^"^ ''^ ^^l-'ools as well as the indicators specific to 
S^e y^rifi.?' K ^ ^"'^^ quality indicatoS 

in^L^^V^? ^"^^ development of tools to measure the quality 
S?hS '?**^yf y communicate the results to the stakeholders. As the dam 
Srl^^^' '"^."1 early siages. the number of indicators included S 
each report casd cannot be detenmned. At the conclusion of phase two. each school 
be provided with the means to gather quality indicator infonnarion on an S 

}y expected that these annual report cards wiU serve stakeholders in 

f k J'SS!!'L ^^'^^ strengths and weaknesses of their individual schools and 
S^f T - f" "'^^.-'^^ ^° collectively plan and improve die quaUty of their 
l^^w iS""^ presentations during phase one of the study have ai^dy piSiuccd 
schools who wish to volunteer to develop report cards during phase tvloofth- 
smdy Phase two of the smdy will also sec*^the>)ject team identify a c^p^bk 
school district to pUot the gadiering and reporting of quSi^indicaSre TOsSl 

?r'r?/ir,T^^' "^I^^^y ^^"^^^ ^^"^^^^ °" theSbeTs of tSs 
process m schools and districts other than Grande Prairie. It will also serv^ to 
^enofy common indicators and disparities which exist from oLThS.1 d?sS to 

Phase Three: Annual Report Cards 

The final year of the Grande Prairie smdy wiU be the pubUcation of annual 
report cards onthe school district and each school witiiin the SctTc foiSat of 

reflect the diffenng opimons of each schools' stakeholders. The metiiodoS used 
to gather and report on these indicators will be formaUzed s^the daS be 

for settmg goals, planning change and measuring pk)gress of each school from veS 
to year Admmistrators and teachers need not wSste valuabl^mc Seri?? dam 
which has no sigmficant value to the district or any other stakehS. Stio^^^^ 
Tof 5 Tl^^'l'^ ^^V'^ of feedback to its stakeholders. DaWd Ss say 
I can t think of any other single sector of American society that has absortS more 

a?9o! p.'26T"^ ^"^y SS^S 

Kemis goes on to say, "Teaching is the only profession I kiiow of that if 
you do well, nothmg good happens to you. and if you do p(^ lv' nothing ^^^^^^^ 
happens to you" (Mann. 1990. p. 26). The statements of LvidTl^s 

SeS'^n P"''"'^ - deaUng°^^^educatioT; 

need ha. been demonstrated for more innovative methods of pilviding stakeiiolders 
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with feedback. Stakeholders can no longer rely on the convenience of standardized 
test scores and comparisons with national averages as the measures of the quality of 
the educational systems. In an article in the Executive Educator, John G. Weiss and 
Arnold F. Fege say, "Ultimately, instead of measuring die success of die public 
education system, standardized tests will dictate what is taught" . Weiss and Fege 
indicate in their article that what is measured becomes a key factor in determining 
behavior. "So there is every reason to believe die introdrction of state by state 
comparisons of standardized test scores will influence the public's perception of 
each state's educational system." These changes, die autiiors believe, will result in 
die public demanding changes diat would evenmally harm schools. The audiors 
state diat the test were never meant to be a vehicle of change, yet diat is what diey 
are capable of becoming. In closing die autiiors say, "Unless we establish 
safeguards, we might find diat standardized tests have become die tail that waes die 
dog" (Weiss, & Fege, 1988, p. 14). 

Educators, must begin to listen to communities and fellow educators and 
report on information that will provide a basis for change in education diat will be 
positive and in die best interests of all stakeholders in education. C3iange is not an 
end product but an ongoing process. The ongoing process will reflect die changes 
m society and die job market to continuously educate youdi widi information diat 
will be valuable to diem for life. Education cannot be content widi die philosophy 
diat prevaded in die past where educators were providing smdents widi loiowledge 
and skills which arc not required by industries of today. 
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A CoUaborative Model for School and Program Evaluation 

Tn recent years there has been a marked increase in the enmh^ 
^grmi and school c/aluatioM. In Lethbridge School District #51, andinS 
throughotjt much of Alberta, the approach used has been piedominantiy 'top down* 
wiih the planning and prtxedures canied out abncst totally by personnel fhan the 
distnrt OTitral ofiBce and assisted by external resource persons. A meta-cvahiation 
ccadudcd Aat hnnied outcomes were being realiaed by the evaluatiooi since the 
teachers and sdiooi-based administrators were not actively involved (except for 

sto^f the rel^ hterature and of procedures foflowed in other locations led to the 
FnTfl^J * 5?f??*^* T^' ^ snpowcrmg of school-based persomiel 
tobe acQwiy and profcssKmalfy invoked in their pro 

lueuseof educanonai quality indicators' serves as a key conmonent of the 
proposed model. 

Introdnction 

'Setioua conversations about education matters must precede the 
evaluation of school organizations, personnel, students and 
cumeulum. It ts through these conversations diatwewiUbe able 
to get our ideas straight about quaUty in education and about how 
to reach it.' (Common, 1987, p. 11) 

In recent years, evaluation of teaching and of teachers has received in-depth 
sm^but httte effort seem to have been put into whether the models md 
procedures wed for the relatively new focus on school and program evaluations are 
consistent with the findings of educational research. Because the stated goal of 
evaluation is ahnost always sdiool improvement, there is a need to consider and 
uW^ the research on qiiaUty indicators, school effectiveness, and school 
idprovcnKnt 

h^n^jfi?^^ °^ ^* perceived need for an analysis of the procedures 

being used m one Alberta school jurisdiction to carry out program and school 
evahiaions. and for the development of a new model based on tiie current research 
and literamre m the areas of indicators of effectiveness, quaUty. and school 

^ programs was sucdncfly described 
by Goodlad (1984, p.l): 'to survive, an institution requires from its cUents 
substantial faith m its usefulness and a measure of satisfaction with its 
perfOTmance. 

Backgramd 

In the province of Alberta, as in other Canadian provinces, and a number of 
Amencan statw there has been an increasing emphasis and expectation from 
governments that school jurisdictions wiU develop and cany out formali2ed 
procedures for evaluation, inchjding the evaluation of students, teachers. proErams. 
schools, and school systems. 

"niis desaiptive research projea invoked carrying out a case suidy of the model 
of school aM program evaluation employed in Lethbridge Schod District #5 1 (LSD 
#51), a mechum-sized Alberta jurisdiction of approximately 8,000 students and 450 
professional staff. The Lethbridge model mvolved the use of a large team (up to 
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fifty five peoplr) Irom the school district central office other sch-v^u aik-^ 
^<W(?rfiK A was extremdy expensive in ter^ 

was produc ng payofGs in professional wowih and dSS^nSmt p«S^^5?^^ 
significant iotcot was «Sth the Pertiaps the most 

p-Z^^^ «accm was wim me problem of miplememmg the recommendatinns 

Purpose ami SigmGcancs of the Study 

to'i^tf w^?^'*"^'?'^* of ewt/Mtfoa is act 

to prove tof to anproiTB.' stafOebeam and Shinkfidd (1985, p. 151). 

In carrying out this stucfy. the folkwing research questi^ 

Lttt^'JflS^H^^^i^lP^^^^^J^Sf^m and school evahj^ 

the hleranire on nxhcaton of cffcctivene«. qua%. and school improJm^ 

^AreAe proccdui«. instnimena. and data used in the LSD ^ 
effiStllfrSS^i?^^ ""^^ (Aaracteristics or cateiia cf roLm effective and 

Review of Related Literature 

of vU^"" SitnH^ ''^'^^ to be the worst 

r984 pTo) ^'''^ 'PP^*^ to be the best' (CSoodlad. 

ti^SdJtt^l^i^oSt^^^ 

evahiatiDn - theory and practice. 
S^^* "^"^^ "f*^' <3) indicators of effectiveness or quaUty (4) 
s^^provement Uterature, and (5) evaluation mSS.^S^'4^ 
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c^wS"2^*'.,nL*^^°^ii^^ program evaluations were succinctly stated by 
Stufflebeam (1971 . p. 157) as 'not to prove but to improve!' Other items from the 
Utcrature revealed consistency in the view that schools and programs should be 
evaluated. Both Eisner (1985b) and Goodlad (1984) stressed tlw lack of pubUc 
confidence m our schools. Eisner (1985b. p.l) suggested that. It is possible that 
our entte educanonai system is near collapse.' Common (1987) noted an apparent 
contradiction between what Madsani magaaae referred to as a crisis m CaSan 
and Amencan schools (Fmlayson. 1985). and the many Canadian research smdies 
and poUs showing that teachers, parents, and students regaid our schools as doing a 
reasonably good job. Common (1987) concluded: There is rightfully a justificatoi 
for careful evaluation of the curriculum, the professionals and students, and the 
organization that constitutes public education' ^.9). 

Meta-Evakisdoa: Evabatiag the Evabatioa 

'Quis cttstodiet ipaoM custodes?' 

The above reference by Morgan (1986. p. 26) to 'who gt;ards the guards'?' 
«iyes as a reminder that evaluations themselves need to be evafaiated. and tiiat those 
doing the evaluations need to be competent and trustworthy. He asserted that no 
mmiune from evaluation, especially not the evaluators themselves. 
Levm (1983. p. 1 1) concurred and noted 'Given the amount of time, effort and 
monqr which m^ he involved in an evaluation, and the importance which its results 
may have, school districts need to be sure that evaluations do 'deliver tiie goods." 

Common (1987. pp. 11-15) strongly put forth a number of concerns about 
evaluation as it is now done. -When evaluation is a public action, it becomes a 
political activity m which power and powerless become central' She stated her 
opposiuon to external models for evaluation because they are costly, contentious 
tower teacha morale, and generate data ^dx may be used very little. She stated 
mrthCT that they promote an 'impoverished view of education and de^ 
at a tune when his (or her) professional skills are more imporant than ever ' Her 
view is that external models of evaluation may stop curriculum innovation and. at 
best, are unlikefy to make it start 

In considering the natore of evaluations, and what should take place. 
SmfHebeam and Shinkfield (1985. pp. 70-74) reported that Ralph W. Tyler 
(generaUy recognized as the founder of educational evaluation) considered that 
evaluation should determine the congruence between perfonnancp and objectives, 
pis approach laid the foundation for an objective-oriented style of evahiation as far 
oacicas 1942. 

hi considering the 'hoW of evaluation. Eisner (1979. p. 267) expressed his 
sttong vww tiiat procedures and criteria used to evaluate smdents. teachers and 
school atanstrms 'have profound effects on the content and form of schooling.' 
He seaned to lend considerable support to the argument for school-based evahiation 
^_!!2f^^V^?' ^ ba^c of educational excellence' (p. 

}u h P- strategic advice when he statol 

mat eucffts at unprovement must encompass the school as a system of interacting 
parts, each afifecting the others.' j' ui mi«.«.uiis 

Scriyen arid Stufflebeam discussed 'raeta-evaluation' and the need for assessing 
evaluation criteria, processes, and techniques in Standards for Evaluations of 
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^^f^J^^'^J^'^'^^^^(^^^^)' Soiven (1976) stressed the 

T^^^ evsauation He emphasized that it could be fonnative or 
suomatweaad could mdude use of the 'Key Evaluation Checklist' tKSiS 
^ "^^^ evaluation involves multiple dimensions and^S 
Si^^mS? mffi^'"' ""^'P^^ ^"^^ °^ measurement, and= 



Effective Schools Research 



'It is vasUy easier to describe effectiveness 
tlian to achieve if Leithwoodaud FWlan (1984, p. 12) 

ni, wS^« "1^**?^^*?^°°^ ^ somewhat in response to the weU- 

SiSS^ works of Coleman (1966) and Jencks (1973). hdd very 

school could do very little to nihice ippar^t 
among children mtenns of achievemenL n^^y 

Ann^°*.^S P'^^ s"«eed where others fail 

^f!5SSl£^® characteristics of successful schools: principals who prov^S 
strmg admmjstrative and instructi^ 

SS^"^ stiSilatS^uS^^S«wK 
^bS^ScSJ^SS^""^ quiet without being repre^ZLSS^ll 
^St^^^ ^^^""P^"^^ and a continuous system of monitoring 

Tlie majOT importance of school leadership was reiterated by Goodlad n979^ 
^0 asserted that the prindpai is the oSSlfiguie inSH^^^Sfa quaUty 
sdiMl program. Smiultaneously, in Britain, Rutter etal. (1979) identified that it 

^S^wS- ^ ^'^^^ id^tSe? the 

J*^*^ ^^^'K finn leadership and teacher 

S SSJ^ I^^^* clima*' «h<»i orgamza^ and teacher 

^SSS.S^^i®^^! appearance and comfort of scliool environment, 
and opportumties for student participation and respcnsibiMly. 

res2S'*^«JS^°^2!, ^ ^ 'effective-schools 

Slft^Si^5?2:i^ " "^''^ °^ «»»«=l"ded that SdS^f 

i^):^s^ssfc=^s&^e?-°^^^«^ 

H.r22fI?i!J^"f®^^''^^^ Shuhnan(1984.p.6) 
J^l^dTTB!^^.,*' *^i^.«i"«tive settingsfor teach^' A four 
Su^'lfrS^t?"*^ Monimoie and Sammons (1987) 

i^SSL?^ ^^^i''' ^"^^ ^ ^ accounted for by 

SS'^^. P"*^«« con^i of the principal and 

^^Xfi^"^ not writing about teaching. Peters and WaSan (1982) 
? l»*-^?»ni»nies were outstanding not because they were able to 

Sl^t^,^!2?°^ because they were able to mcJttvate average 

employees to oaraordmary dedication and performance 
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r-f? »«°^cant interest is the observation from the Uterature that not aU 
rwearchers and wnters on the topic are conunitted to the effective school 
phflosophy;anumberofcfiticismshaveemerged. CSlickman (1987. p. 624) argued 
that schools and rwcarchers have failed to distinguiah between good and effwtive 
schools and that The 'effectiveness' movement is unnecessarily restricting the 
OTO^um. narrowing the teaching approach to direct instruction, and conoSling 

• J^^P^^ ^ identified problems with the effective^chools research 
mdudtag that no one knows how to create effective schools, the language is fu2zv 
effectiveness is constricted to test results, and most research done hi 
elementary schools. Stedman (1987, p. 222) concurred and argued, If we 
contume to evahiate sdiools with narrowly designed standardized tests then we will 
get nanxjwiy structured schools to impart such skills.' 

Stedman (1988, p. 442) advocated consklering an altemative approadi based on 
cultural phirahsm, a^denricaUy rich programs, personai anenlion to students, and 
steml govemmoe with parents and teachers. FUlan (1985, p. 414) cautianed that 
Z°£!5" ^ ^ establishing a grand scheme putting all schools in 
the district tnrough the paces of developing effective school plais. ' 

It is evident that evaluations of programs and schools must took well beyond 
the cntena aMnmoniy associated with the effectiv^-schools movement Evaiua:ioos 
must be hroadly-btBed and muhi-dimensiooaL 



bxUcatocs ofEStectiveness or Quality 

c / °^JL!^°^ literature is beginning to evohre and offers promise as a means 
of descnbmg effectiveness and quality it educajion. The development of 

mdicators' is providing a new focus and emphasis in educatiooai studies and in 
schools. 

Common (1987, pp. 10-11) described quafity in education as a 'mental 
construct,' and daboratedfliat ^ 

'What we deem to be quafity in education and the criteria we use to judge it 
IS as much a product of the moraUty and other vahies of the times as it is the 
product of what we know about what ws are doing. Hie secrets of what we 
deem quahty m education to be lie notin the measmwnents we take, but in the 
judgements of worth that we make about those measurements.' 

Mumane and Pauly (1988) stressed the importance of developing multiple 

^Z^!^' ^^^^ Pojn^ indicators provide 

information about the health of a school system but cautioned that a common set of 
o ?f?°!? ceitamly mcrease the move toward centralization. A 

u ^ Coleman (1986) introduced an element caUsd 

school/distnct etiios' or goodness as a potential predictor of school effectiveness. 

In the view of Porter (1988, pp. 504-505), ft? use of indicators could lead to 
more centrahzed control over the education system in terms of what, how, by 
whom, and to what standards things are taught His pessimism was offset by the 
sratementthat designing and unplementing:^ system of educational mdicatois could 
become one of the first steps toward the meaniiigful participation of teachers in 
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setJmg educational policy' (p. 506^^ Porter fotmw the possibility of enhandmz the 

^ but was also cognizant of the danger thatpartidpaticmTthe developm^ 
^ooaa^ could result in . . . the lowest common denootaator of current p^a' 

n I?^?^'^.?®^^f^^o^Education(I982)pr8scntedm^ 

12 categones with a total of 42 indicators: cuiricular congruma. SSwT 

SSfc'^ P^?^' «P««ations. school-wide nS^^^uS^^ 
Tt^^' "monitoring and feedback of stodent Sop^ 

SJJf^?!h!^j£?*^^^'^ST"^*y^^°^^«^^ This list bean a striking 
^SrS^lf^^"^^ earlier in the review of Uta^ 

and should be of considerable use in meeting the purposes of this sttidy. 

the^J^,^^ *^^*^ C^umbia considerable efforts are underway using 
f^Q^'^J^^^^°^ofqaa&ty. The British Columbia Mfanstiy of EduS 
i , Jf^^-^H-^ P««nted anevaluationmodd of thrtc conqxfflenls:^ 
qujdxty md^catOTs. and an interpretive framework (ftcjudamwnSlmd vtevraS 
go^aonab aadmembeis of the public who 

°^ 5^^' and beihavioral o^~ 

^J,^Sf^»"5^ standardized tests, program participation, diploma type. 

SSrfaS^S'^'"'*";v'^^'"^°^P»- ffiecSe incticatSis inclSSd 
SSfS^^iiJ? f5°°^.?' !£Ff»f««n. motivation, values, and attitudes toward 
SS^!S ly*-^?"^^ .T?ior indicators of bdiavioral outcomes were in 

*f^* TJeindicators of physical behaviors were fitness. 
^ fr««lo«n from substance abuse, social diseases 
pregnMcy, andsmade. The indicators of social behavior included the set of 

S^t,Sf^?nli^!!f^*? ^^^^ government of Alberta - 

etiiical/morai, intellectual, and soaal and pwsonal. 

The MMstoK de I'Education, Gouvemementdu Quebec (1989), published a set 
S^SST °' responding to the demand for accountability in pubfic 
adnmusttatioiL IheirindicatDra were presented in five categories: 
hnard JilSSfjl'Sl""^.' including spending in relation to GNP, school 
^^f^P^"^*^ and average tea^ 
, il' "Egress through school, measured by numbers reachine and 

t„««KJj*lS^^ graduates in terms of numbers and types of diplomas. 

numbera gomg on to coflege, and numbers joining the woikforce 

gradual «*i««^ inchjding spending by boaiti, and numbers of adult 

In the ynited States of America, The OfQcc of Educational Research and 
tagov«nr^t (1987) presented tiie foflowing 'outcomes' as indicS^^ 
perfonnance. college-entrance examination scorcsVS 

^ ""^"^^^ ^ of yo^ng adults, aS 

FfftopatiOT of high-schcol graduates in postsecondaiy educatitk ffis 'resoiirces' 
listed as indicators were expenditores per pupil, pupil/teacher ratios, and teacher 
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salaries. The contexf mdicatois were school emoUment (by age groups), aspects 
of ho^e eimroninent, student drug and alcohol abuse, teacher job sSrfacSi 
school problems as seen by teachers and tiie pubUc, public opinion ratines of 
schools, and stete high schooJ graduation requirements. 

Sraecautiom are wan^ted by the indication 
SlJS^/i?^^?*^^°'^^«hoolq^<y>ndica^ She 
warned iiat the quality ndicators axe uninterpretable, and can be n^deadins It is 
me view 01 tne rwearcher that this need not be tn:e since the indicatore are 
developed by teachers and should be written in such a way that they can be 
mteipretsd easily. ' 



School Imptovement 

'CHiange is a process, not an event!' 

(Hall and Loucks, 1977, p. 17). 

Since the goal of school and program evahiation is the improvement of schools. 
It is imporau: to consider the Uteratore on improvement Qose similarities exist 
betwee:? tni rv- here and to the research on effective schools. 

Leithwood and FuUiu: (1984) proposed six strategies for increasing the chance 
of suaessful change: continuous prof<-5sional development, increasing principal 
ettectiveae,w. school planning, developing poUcies with a view to their 
unplementation. using standard, operating procedures, and building systematic 
problem-folvmg procedures. j-a^iuauv, 

Fullan (1985) went even farther and presented a set of schooHevel strategies, 
u °f a plan, investing in local fadUtators. allocating resources. 

SSSf- °° «=°P* °^ P^'j*^ developing the principal's 

leaderslup role, focusmg on instruction, stressing ongoing staff developmmt, 
ensuring mformation gathering and use. planning for continuation and spreS^and 
reviewmg the capacity for future change. api«»u.<uiu 

Lezotte and Bancroft (1935) noted that successful local school improvement 
programs have in common a focus on a single school, a buUding-based 
improvement team, a longer-term orientatian (three to five yean) in planning and 
unplonentation, and are organized around the concept of the effective school as in 
research. 

i^ZSi'Jl'^^'^V^'^ P«*»«t *ru« for school 

improvement differs from the past in that the target is no longer the district or 
mdividual staff member, but the school. Their conclusions were that the primary 
method for achievmg improvement is not curriculum development but staff 
developmrat. that ttie source of improvement is not just intuition but research on 
effective schools and effective instructianal practices, and that planning is no longer 
year-to- year responding only to immediate needs, concerns and problems, but is 
proactive, long range, and systematic. 

Naisbitt (198? p. 3) appeared to lend support to this school-based model of 
improvement witti his statement Trends are bottom-up. fads top-down ' He 
advocated moving away from the specialist u'ho is soon obsolete to the generalist 
WiiO can adapt to a liigh-tech/high-touch' worid. Although he was speaking 
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mamty^ut busmcsses. Naisbitf s ideas, long-range plans must replace short- 
tenn pront' could apply equally to schools 82); he declarS ^t'^Sc 
planning is worthless - unless there is fim sStegic visio^ 94^ wSffi 
^ that -Td^pwcrsareate leader. Period/ (p. Sl)VSS«SesSe dS?S 
7?^; ^ Saskatche«^ Ministi's AdVisoiy Commit 0^85 n 

LJty ^c*^^* ^"'"^ «*«>i pla^inglmdTctiai mi£t 

occurwithm. 'School improvement is about taking actiofSSeloSle^^ 

In ccmclusion^e school improvement literature has presented several 
proportions and strategies for school change. TTie consistent S^on SSd 
IS mcorporated into the section of this^elSSS 
gop^^a prototype for school and program evaluation for LethbSdge 

Evabatian Modds. Paradigms, Cnteria, and Standards 

In tiie oinnion of the researcher, the most significant debate related to the best 
S^ltiSit^;^'?" cxtemal'^SU inteS? fS? l^^dTc 

Skr^J conducted by external evaluators or be done at the school level by 

T^^^tJ^t aiyroach having the most promise is one that wiU seek to cultivate 
l^^y °^ '"^^ °^ I^biems. to become larg^ itf 

^^rt^^lil^' P'f ? ^"^^ supported the school-based approach: The 
^A^lfH^fv.^*" °^ "iu'^ational exceUence.' Good and BrSy (1986 p 

^ °° evaluation mScb ^ stotog 

«,«2^f!f!?'* <^^*^> of both the 'topKiown' and the 'collaborative' 

Utt^y to be based on a collaborative approach since it assures acceptabifity which 
^l^J^°^^°^.^y'^V/BndTtMmy. Hestatedapi^eSSSn 
ZfCJ^ muror miage and complement of perfonnaS^a^rS'^^ 
^i^^l demonstrated by appropriate and Sfective^ of 

SSSSZ??"^"?'^^^^"^**^^- A?a^ofcautioihett^' 
e^^ fo^^«^ obsen^ations. there should be wSSIS 

SS^L of appraisal mcludmg assessment of the diagnosis, planning 

choice of st^tegy, choice of materials, and methods of teaching^^ ptamiing, 

beca^'^nl^SJi? ^^v«^J«i strongly against external accountabiUty 

^^JmafSSiSf r^Sf^ VS^T"^ ^"^^ ^^^^ «»iKht be used 

SvaSS^ cause Curriculum 

self.Sv^ohSL**^n,^^>?"'^^ ^ P'*"^ of school evaluation: 
SvilS SlJf * tfam-visit/reporting phase, and implementation phase. He 
f^^^^}}°^!^ool staff members could effectively utilize the^setf-stodv 

(kwlopaplanforochoolimproveaent Shaw believed that after a thoro^seS 
-stody is undertaten and accomplished, visiting team mem^ STc?ve as 
external vahdatorsfor the WD±ofthelocalstaffZcaSoSSS 
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shouidreco^ize Aey camot leam as much during the three-day visit as local staff 
member* already toow about their school program. The study reported findings 
comparing the ev^ve judgments of local school staff members with thosv- Tf 
vositmg team members; 13 of the 19 items showed no significant difference five 
showed signifeant deference, and two items were rated higher by the visitins'team 
(related to student behavior) while four were rated Ug)aer^]oc^^^S^ 

Novak (1985) asserted that too much stress, time, and money are invested in 

^?JS!^hlf SSSS"^ Ml!!flT°^^"^ ^ evaluations, even though 

a«reemgtlwt schools could Thisw(& 
suggested broader involvement of the educational consumer in the design and 
opjOTQon oftlteevahJatianprooeas. He advocated shortening the external viStetion 
while acknowiedgmg staff insecurities if there aro insiScient visiting subjert 
specialists to cover every discipline. 

An Australian model by Boud and Donovan (1982) documented a set of 
prmaples to guide the practice of evaluation: teachers are the people who have to 
implement the changes, dedsion-making is devolved to schools, internal evaluation 
replaces wttemal evaluation, and all members of the schod participate in the 
process, fioud and Donovan concluded that participation in the process was just as 
impOTtant as the implementalion of the changes, and teachers reported that much of 
the benefit was as a result of their invohremem in • 
the {banning and conduci of the evahiation. 

Herman (1986, p. 3) suggested a model involving a 'top-down, bottom up' 
approach which is srhool-district based but oriented to meet school building 
cli^room, and stete needs. She noted the main problem with existing top-dovim 
models was that fte people at the bottom (teachers and local admhiis&ntQra) were 
t^Z^^tF!^^ than data users and that paperwork and bureaucratic 
^^i^ ^ supported, school operations and improvement 
efforts. 'Bottom-up' ncrJs were not being met ^ ^ 

WUcox (19«9, p. 188-189) noted in her British Columbia study of school and 
pro-am evaluation models that 'although their opinions about a program are 
^ stakeholders have control over the direction the evahiati« takes ' 




In Bntah Columbia, the Ministry of Education has developed procedures for 
'fT'* program evaluation function since 

acCTKlitation is defined m the Accreditation Booklet for Secondary Schools (1983, 
p. 2) as tlK outcomes of an internal and external evaiuatioa' Internal evaluation is 
undertaken by the staff and administration within the school and is designed to 
mcomgeand assist m the improvement of the school by its own mtisSvt and 
eilort External evahiation is undertaken by an external committee and is designed 
to provide mi evahiation in a broader frame of reference to confirm or question the 
internal evahiation (p. 2). 

A few Alberta school jurisdictions place significant emphasis on a school-based 
model of school and program evaluations. The Spirit River School Division #47 
employs theB.C. model of internal and external evahiation. The Edmonton PubUc 
bchool Board employs a school-based decision-making model which also appUes to 
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schTOl and program evaluation. Edmonton utilizes school-imtiated and school- 
ba$«ievaluauon by staff to assess schod-based pro SuchevaSoS^ 
d«i^ed, admmatered, and analyzed by school staff with assistance (on request) of 

SS^tJ^** ^^\. self.awa«neti invtltories to 

c^their understandiDg of the program and prepare them for their discussions 
^^IzH^ classroom practices, and to prepare them for classroom observations 
^^^^^f^^^- At the distria level, they have produced outcomes and 
sS^f mathematics and are working on them in other 

^ini^ti^- ^Ptnculmn is defined in terms of inumdcd outcomes, 

expectations. andsamiJeradicatDrs. District achievement tests have been developed 
SSi°°T?^ti:^' ^' ^2 ia°««age arts, mathematics, science, and soS 

i^^^sirde^c^^^^'p^^^y^- 

stB!^^^^^^.^^^^:^^.^^'^ (With cooperation of 
S?,!?^ 'rifPf^ts of the school dmiate and operatiooi. employing school self. 

^JJ^SSlfff^-^^SIP^'^^^ District-area 
J^S"?* adf ^vahiation activities which have been 
put together mto a 'School Prome'- a framework to help document and assess 
piwempractiMandsttprioiitiesforfutu^ Their apprwuJi is based upon 

p^ni"JS.^i^';i?^ dooibcd in a document from the aty of SalforJ 
gglai^d, &fto^ Lookwg at Thaasdws (1963) and a puhHcation. Xeqanv the 
^^it^ by Casey and Malion (1982). It is groumled inSTbasic 

^mmptions that school evaluation is a continuous process by which the school sets 
^ ^^^^ the parameters of the prxmndal 

School Act and board pohdes, that involvement of prmaoai and staff is 
fundamentel, and that most decisions should rest with them with the school scckine 
external advice and assisteoice as it judges ippropiiate. 

JTiisrwearrfier. after a thorough review of the Bteran^ 
m education and evaluation, oondudes that the dedsicn en the debate related to the 
exteml versus mtemal evahiation model falli dearly on the side of the school-based 
vi^LnoS^T^ n^^?^^''^ ^<^an (1982), Common (1987). 
SS!fSi 1 ^ compelling reasons for a 

modelwluch mvolves more active and professional partidpationof sdiool-based 
posomeL Theen^wennemof sdiod-basedteadienandadministratois. and the 
-chool as the focus of action and development offer the greatest potential for real 
growth and developmem of the sdiool. its staff . and its programs for smdents 
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Research Methodology 



IWs rweaich study, a meta-evahjatkm, utilized a dcscriptive-deaign, caae-studv 
«pproachsmce it dealt with matters primarily quaUtative in nature, but also used 
some oJirelational deign features as required in anafyzingcerta^ Theviews 
of Pattpn (1980. p. 40) who observed that 'r«earchess using quaUtative methods 
stnve to undCTstand phenomena and situations as a whole; evaluators i«ine 
^togje methods attempt to understand programs as a whole' would support this 

Stake (1978, p. 5) observed that case studies will often be the preferred method 
of lescaiTdi since they may be 'epistemologically in harmony with the reader's 

no5i?"*?n!S?^"*^**^P*^*°**^^"«*»^o^ Patton 
(iy»u, p. prescQted a strata for carrying out a case-study and a format for a 

report outtoe (p. 340) which has been used in this rtswffl^ Guba and Lincoln 
(lysi, p. 376) presented a rationale for doing a case study rather than the more 
conventional techriical report for a naturalistic cvahiattoL 

Eisnw-s views ( 1 985a, p. 358) support the approach utilized in this Lethbridjie 
stuay witii his statement that The model of natural science on which educatiomd 
reswch is based is probably inappropriate for most of the problems and aims of 
teaching, lewmng. and curricuhim development' He advocated that educational 
oonnoisseunhip (the art of appreciation) and educational criticism be 'not limited to 
the art^c dpscnption of events' but also include 'their interpretation and ^apraoal' 

The literature was clear and convincing that for the type of naturalistic research 
being donem this study, a quaHtative c^-study design, supplemented by some 
limited statiaiicai analysis, was the preferred mode for the probtem identified. 



Sources cf Data, Data AnatysisPnxedaies 

The notion of 'grounded theoiy as emerging from the bottom up (rather than 
from the tcn> down), attributed to Glaaer and Strauss (1967). and reported in 

P- 2^>' ^^^^ ^ *c -'isparate^eces of 
OTilected evidence that are 'intenxmxted' The Lethbridge stody mad^ 
data ance large quantities of information had been coUected in the past three years 
cL program evaluations, using instrumeals designed by Lethbridge 

viyjw). ineLcthbridgeresults were compared to the findings of Goodlfid. 

was employed for mter-suijjec&ve vaMty verification. This panel coossted four 
scnool prmapals who had their schools evahiated using the Lethbridge model 
They were asked for their asseamient of various aspects of the model and 
procedures which were followed. Their responses were compared to the 
assessment made by the researcher. 
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The data coUected from students, teachers, and paienls were analyzed to check 

u)^)ared with data from the Goodlad (1984) study. The dala-gathe^ 
msminients w«e assessed. ITie Lethbridge model was analyzed and^itS 
?^!SL°^ findmgs from the Uterature on indicators of educational qualitv 
SE!f?if*^*^'*^^^ "«?rovement, other school and program ev^uatira 
research, and the statistical anaiysea. ^ cuu«uua 

frJ^^F°°^ ^ 'P^^ experts', along with other data and infonnation 
S.- ?t"dy. were used in a procedure of tiianguiation to address the 

^araqr of the data and to ir ^ some candusi<ms on^modd cur^^TSr^ 



Findmgs (Analysis and Meta-Eyalnation) 

M«d fn^'lf?"^^'?**'^*'f.*''^"^*'" • • • ^yoivcs both deteriptioB 
and judgment. ' (Guba and linooto, 1981. p. 380) 

oth.^ JSSSJ!^ rclationsh^ was found between the inf onnatkm ofatamed from 
,rS>« i ^ evatoation process and the condusions rradied based 

S2? '^^^^ °^ P"^^' stoticnt, and staff surveys ^ 

S^Sf^!;?^ by eachffoup (parents, students, and teadiersm tSLe «^ 

tl^ifSS^^if^ from thoae in the Goodlad study may be accounted fe^ 

^ countiiesW different sodal and 

SSTIS^S^^?- pie wtocation system in Canada has not suffered to the 
Sfte l^iSs confidence and support as has occurred 

correlation was computed on the ratings of the 
roea^er and the Panel of Experts. The correlation^ computed to be aewS 
^75 ^ ^^^0"^ according to Morehouse and StuU 

the pSiS^lJir^* ? greatest agreement between the researdter and 
me mel of Experts. The reaeardier and the pand gave highly positive ratimis to 

SfSS'T^r^^^^*^!^ -nTesemffl^JScS 
™^o!!rS^°°?*°^ opportunity for parental mput, recommendations whidi 
^^f!?^^ .accurate^ evaluation report whidi was presented in draft 
d^l ^ Pnor to Its finalization and release, and the f Jet that the sdiool 

Simiiar strrngly positive 

SST^^StSLS?^ predominant!-, 'top down', 

^ft Ai^2^»Sf!^S''°^''*^°'^°^^«' sdiool-based administratDns. 
S !i«^5J^?J?°^,H^P^i to a^eation paid to school 

soaal mputs and to the danty of the evaluators' role after the leport ^written. 

The researcher was much less positive than was the pand that the input of 

agreement existed between the researcher and the panel in a mimberof other areas. 
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The usefiiloes of fte evaluation report was rated fairiy hi^ 

The approach to school and program evahiation as used in the Lethbridee 
models encon^jasses virtuaUy every one of the indicators of effectrveneas presented 
^iS^ in Aeir questionnaire for assessing school and dasstoom 

tfr«aweness. The Lethbridge models appear to meet Goodlad's (1984) assertion 
mat efforts at school improvement must encompass the school as a system of 
mteractmgpameadi affecting the other* (p. 31). Hie broadly based emphasis of 
the models with a focus on aU aspects of the school ranging from insSctional 
programs to non-msttijctional programs admowiedges that each pait of the school's 
operatKOT affects all other aspects. 

^ The UAtedge model adequately addresses maiqr of the checkpoints of the Key 
evaluation Oieckhstdevdoped by Scnven (1976) m terms of description, clients, 
lunction, consumer, process, outcomes, generalizability, significance, and 
r^ortmg. mtiie view of the researcher and confirmed by the Panel <rf Experts 
there are some problenu wife the delivery system, the standani^ 
are evahiated (the lack of a clear set of indicators of educational quditor) the 
usefulness of the outcomes (since there are problems in implementing 
reconmiendations).iheco8ts,andtiiethQroughnessoftiiemeta.evaluatiS»^ ^ 

Theladccfabaototestandaidsby^jiiichprogramscanbecanqsaredandasae^ 
IS a weataeas of the Lethbridge modds. Another area where the Lethbridge modds 
are weak, aaording to The 'E Standards' as produced by Nor& Central 
AssoaatiOT Commissi<Hi on Schools (1987-88), is in teacher involvement in 
assesamgfte effectiveness of the program and plannmg for its in^mrement The 
^es^s^oolewluation procedures in Lethbrid^ place te^ 

me ^ocesi and of fte recommendations for diange witii Htfle real and meaningful 
mvotvanent m deteimining tiie natiire of changes desiraWe. Teadicr involvement 
comes mto place m the present models only after tiie external evaluaton have 
deteimmed \<*at changes should be made. The present schod evahiation models 

BKaude procedures to diedc tiiat tiie diaracteristics common to effective sdiools are 
m place. 

of clearly established and accepted indicators of educational 
esertiv«ieai or quality is one of the major weaknesses of tiie Letiibridge approach 
toschod^mluaticn. Although many of tiie qualitative and quantitative incticatois 
may mfencd in tiie Letiiiaidge models, Aey are not qjedfically identified and 

part of tiie evaluation. Uttie emphasis is placed on interpretive 
ind icat or s of context, input, and process, or on outcome 
mdicaton of a cognitive, affective, or behavioral natiire. A set of standards or 
oases far comparison is absent The result is tiiat judgments are made about 
^^mquality and effe<alyeness witiiout tiie stiragm of a set of agreed-upon 

The literatnre on sdiool improvement leads to some criticisms of tiie schr d1 
evata^nmdelinquotion. The concerns ofLdtiiwood and Fullan (1984) have 
not been ad^essed adequately. They believed tiiat successful change involves 
pieasiTC gradualfy acquired tiirough interaction witii peers and otiier leaden, not ^ 
nnposed laessme mandated by autiiority. The present evahiation modeb place 
cmphMB for change from tiie pressure brought about by tiie evahiation report 
instead of from interaction witiipeen. Similarily, me collaborative planning as 
espoused by Patterson, Purkey, and Parker (1986) is limited and restiicted by me 
present models. The cuneat LSD #51 approach does not utilize tiie suggestions of 
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Landon and Shirer (1986) in the Wisconsin School Evahiatkm Plan tn h«v» a. 
sdiooi conduct a sdf.«raiuation .vhich is then S 

inst^ite^Si^^ a focus on staff development 



Sommary, ConclnaioM, and Recommcndatioiis - 
Reyising the Model 

Trends are bottom-up, fMds top-down. ' (Naisbilt, 1982. p.3) 
forSjJ^f^*"^?^^.*^^ model 

SSSf <J^f * ^« °^ P^«««o and admimstnrt^e ^Sb^ 

^SSK^f ^'^'^"^ .particqation and picfesoaul ff5wSl)y those ?*o 
^ rftSfS^***''^°°' **«*~l»^andadniSstra^^ 

^tte own programs and then planning f o?^^St TfSSS 

stuSt '^^t ^ ^ examination of a broad range of 

S^ti^S^?h^^'^°"' achievemS^^^i 
^Sal Sid^in2?,2f^ f ^ of the school encompassing both 

whethM.tkrj«*v!Lz^r^ • Some uncertainly exists as to 

^f^^ from the model is of suffidcnt vahi^ in^ of 

andeaerev ^luvement to warrant the cost of human resource tnnc 

Bcc^^SiSS^^^^ """^^^ are the lack of a deariy established and 
tiS^^^S^;?^ mdicators of quality and the pcoblemi in implement 
SlSS^o^^f^ members of the district Central staff^ehS^ 
S^SKeSSf ^f^"^^°"- T^^thotougfanessol^^SS 

tS^^'^^^.if- ^ ^"^^ ™^ manr erf the Criteria of 
enectiveneas, quahty, and mipiovement in terms of recciviDg input from all 
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satehoUers. uQhmig anc^^ 
mutadmensionalfoa^^^ 

instoooionaiaspectsoftheschooL Hie modd was judged by the researcher to have 
limgtons m that it was 'top^own' with Uttle^jportunty for mea^X 
participatiDa and piofearionaigrowm by theses jben 
are doubts as to whether the infonnation gained from the model is of 
suffiaratjmhje related to pn)fessionai growth and improv^^ 

Amajorweakneasofftemodelis 

Cmicerns with the abihly of sdiools to impleme^ 

senous questions about whether significant inqxovemcnti wiU come about 

This case»stud/ determined that the prxedures and hBtruHCT 
taa, content, and comtructvalidily. Noattemptwasmadetodemcaistiateextesiial 
TSSL^'!L^* ^ to generalize the condusians reached at one 

school to another Some doubtt about leliabiBQr and validity were ndaed since 
tte Items WOT not fidd tested and some 

with loss of comparahdity of certain results. Although reliability (internal 
conastency or stability) was never calculated, the instruments appear to be 
consattttas uKhcated by the high cotrelationa whkdi were found. 

Tte Hteranne was dear that ftere are certain criteria and diaiacteri^ 
Muidbe mcorporatedinto the Uthbridge school eva<uatkmpnxedures to increase 
effectiveness and efficiency: 

1. The new modd should move away from the heavy emphasis on a 'top- 
^wn a^Jroach to encourage and empower school staff and adnrimstrators to 
be activetymvohred and interacting in aasesnng tiie rffectiveness of &eir own 
pogramsandplannmgfortfaeirmiprovemenL A modd hwohing a better 
l>alan«b^eeninternd and externdevahiation should be consklered. School 
-based subject-area evahiation by intemd sub-committees, followed up by a 
review by the external team to confirm, question or add new factors to the 
assessment is recommended. 

2. Tlie development of a dear set of standards or indicators of educational 
quahty would be a desirable step in moving the evahiation modd onto more 
objective ground, and would allow, perhaps even demand, much more 
professional mvoh^ement of school level personnd. Indicators should be 
developed by school-levd committees of teachers and administrators, and 
siiand with and revised as necessary by other schools. The process of 
oevdogng educatioD alquaayindicatoraisan evototiooary one; the result, after 
^If?? ^proximately three years, should be a district-wide set of mdicators 
whidi have been teacher-developed and validated. The indicators may vary 
sanewhat b^wen sdioob and particularly between levels or divisi^ 

into accotmtthdrmiique features. Indicators should consider the school sodal 
i^uts such as tiie smdmt body con^xirtion, and the schod sodd stn^ 
ctoaiB, and should focus on smdent outcomes in the cognitive, a«ective, and 
MmviOTddon^^ 

oclSv ^ °^ McEwen and (1989), Shedlm (1986), and the 
^Jboia(1989b) and British CdumWa (1$|j6) papers on qualily indicators have 
produced numerous examples of quality indicators in the various domains. 

3. Evalu^ ^ ^ teadiing staff and evduatkm of the prindpd need not be a 
part of the school and program evahiation process. The prindpd should 
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assume M responsibility (witii assistance and delegation as reouired) for 

and profeaanai expertise shouW n« be so heevify m 

^^^.Z r^"^ that they it be avSS^ most 

^ilSJ^ aJ^^ b« P«t into meta-evaluaticn 

^SS!?^ evaluated by all slakdiolders 

TliebouxiofedBcaticS 

meta. evaluation m terms of whether it producedleceSSy md^^ 
^onMtxon. Rdated information on the value and^^c^ ^ 

«*ajls who wiU make use of these (or a 

™^tion<rfth^mc&^ 

ThB win assBtmrefinmg the ptoaas of developing quality!^ 

^'^^^ *51 should be an ongoing process 
^^^^ M ^^'S** «^ five to s^^TtS 

aady«alpait(rfftecanicalumandias«ructioncyde. Each school in LSd5Si 
SSSS'S^ ^ P^^g^-^valuan^coS^SS^a pS^^^^ 

L!5*Sl!i°¥ -^ ^ program evaluation should be happemng 
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Proposed CoUaborativc Model of School EvainaUon 

evaluation is proposed. The diagram (see Picnre n rnntai«. o 
collaborative modelf or the evalSX^f Srinc^Lte Ke crSS 

-ig^^J^^^^ TWs collaborative model places much 

^^S^^^ ^ school-based adminSSS 

S participants and dedaion-makers in i^^^sdZ 

SSS ^cSSr^ °' ^ pn>fessioSrK^SS^ 

^l^.^S^IStS?'^^:^''^^^ The school is the primary unitS 
d^fflojMMJmg^^ 1985). If changes are to occSTSSt^S 

owraers^ that comes fem the opportmnQr to partial 

SS?^^55E???F?**^<^^^ 1982). SdwSwatoation should bri 
aaw^j)wktDwardthi'jendrffteconditionsaieri^^ Schoobdiffer 

and adawwJedged in the evaluation and improvematproceases. 

is fundamental to reviewinfi 
.an empowered school-based s^^SSSSSS 
SSr, SJIS^^" «^ prafessioni decisions rc^S 

meiroNraprogramimprovementandprofesaionaidcvtAfflnient Theentire^wd 

Sto^L^^^-rtS?'^**, ^^^i^* P; 8) supported this view of school 

SS(Sf^'tn2!JSfJ*^ •* ^ g""*? responsibifity to make 

c»^div«andmf<med deci^ He elaborated by staling tlMt'super4ion must 

Speaking of teachers, he stressed that 'without choice and rcsponsiS^ ^evwm 

ar« ^S^JSl? fiteatoe. «Siool improvement and school-based management, 

^e^^5;,S^^^2!*^'P-^^- She asserted that 'SS 
S^^^JS2?^^''=*?*^*^<=~'«n«ceoffl^ If 

SvS^ed^^i^ tJl*«Si°**".*^ ^ ^8°°^ ™» researcher b 
SS^^^Si'T^ '^l'^ experience in the field, that the 

Sri^^ST"^?^^ ^JFTOPch is the one most likely to bring about the 
desmsi staff, program, and school development 
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The Lethbridgc EQI Project - Fntnre Directions 

e i.^*rE"**°^ °^ program and school evaluation as used in Lethbridge 
Sdiool District #5 1 was evaluated according to recent research findings on schc»l 
etrecaveness. qualiQr indicators, and school in^vement, and according to theories 
and practices of sound evaluations. It was found to be adequate in many of the 
cntena of eff«aiveness. quaUty, and improvement, but to be insufficient in a 
number of other important ways identified from theories and practices. The 
propowi collaborative model adds a number of features which should empower 
school-baswi administrators and staff, strengthen the evaluation processwTand 
marease schoo improvement and professional development and growth. The 
proposed ^ implemented on an experimental basis in Lethbridge School 

I^stnct #5 1 . evaluated, and modified as necessary to strive toward the acWevement 
ot sound program and school evaluations and the ultimate goal of unproved 
educational expcnences for students. 

Based on the literature and on the results of the case-study, anew ajproach to 
sdwd and ^gram evahiationis underw^r in Lethbridge Sdiool District #51. The 
Lethbridge EQI project involves impIemeotaticD of a collaborative model of 
pro-am and school evaluation with emphasb on the devetofmrrt of educatkm^ 
qualilymdicatMS. Key components of the devdopii«m of educaJional quality'' 
mdicatora are the collabcration between schooi-based and district-based staff and 
titeresuWag determination of standards, and targets for Olpn^iams 

ni«e eduamonal qua% indicatora wm focus on student outcoflM in the CO 
affcctive^d behavioral domains, and will be devdoped wifliin m interpretive 
Qameworic which places emphasis on context (fte bttis for mterpceting the 
vanatwm among diffacnt groups), inputs (the number aixi type of resources 
available and allocated) and process (the actions or operations whidi iwult in an 
outrome). Data from surveys of students, teachers, parents, and the public wiU be 
anafyzedand qualily indicatora established in terms of compaiatora, standards, and 
t^gets. The result win be each school evahiating its own ed»!*aliou'a quality and 
eftectivenws in a professional manner which serves to empower s6;ool-based staff 
and contribute to meaningful and effective school growth and impioveraenL 
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School System Review: 



A Comprehensive Process 



This paper describes the cooperative development of a comprehensive school 
system review process based on student outcomes as well as the inputs and 
processes that contribute to those outcomes. The focus is on developing an 
assessment process for the whole school system to meet two needs: the demand by 
Uie public for accountabiUty, and a desire by the school system personnel to 
oemcMistrate their ability to improve educational programs. 



Introduction 



The County of Lacombe and the Rocky Mountain School Division, two rural 
school jurisdictions, have had an interest in the development of an educational 
quality indicators system for some time. Because of their concern for 
accountability and school improvement, both jurisdictions hav- since 1985 
conducted student, parent, public and employee surveys on their satisfaction with 
the school systems . As a result, when the opportunity arose to become mvolvcd in 
the provincial imtiative on educational quality indicators, the two jurisdictions 
iubmitted a successful rroject pivpcsal to develop a more comprehensive review 
process based on a quality indicators systtm. 



The project was 'oproved in April, 1989 and will consist of three phases over three 
years: developanent, field-testing and implementation. The first section of the paper 
presents the background to the de>'elopment of the comprehensive review process, 
including the purpose, outcomes expected and related Uterature. The second 
s«:non descnbcs the conceptual framework and the development of the componci-is 
of the review process. The final section describes fixture developments and makes 
some observations about the project 



Purpose and Outcomes of the Study 

The purpose of the project is to develop a comprehensive school system review 
process which can assess the relative health of educational systems by focussing on 
both student outcomes, and the inputs and processes that contribute to those 
outcomes. 



The development of this school system review process will result in- a clear 
statement of goals of student learning; a system of quality indicators for all goals of 
student learning; an mterpretative process that will relate context, input and process 
yanables to the outcomes achieved, and a plan for enhancing the strengths and 
improving areas of weakness. AU of the components of the review process will be 
developed through the full involvement of the educational panners, thereby 
enhancing suppon for the project 
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Related Literature 



As pubhc demands fOTacrountabiUty and improved educational productivity have 
T^^:f!?^J^^^^y^^P^^^ educational quality indicator systems to assess 
tr^^ ^ P^^*^ ^ national educational systetrThS 

S^.^' '^"^ ^ proliferation of articles and papers on not only 

^J^^n ^ apFopnate indicators but also providing the presses on S 
coliectton, analysis and repoiting of findings. 

papers about the development an . implementation of quaUty indicator systems and 
tiiose p^ th« analyze the inputtSnd processes t£t affectStlSg A 
cross-section of papers and studies includes: smdies on Sput. pi^if ^and 
outcome mdicatois,data<oUcction, analysis and reporting. nieL»nd^ ofAe 
review provides a nmnber of meta-analyses of sevSShiidred smdTes ITf^S^ 
that affect student learning in addition to specific studies. 

'^.^M'f^ are generally defined in the Uterature as statistical 

STe^I^ SSi*^ ^ * °f educational^ teS 
usuaUy expressed as a number which can be compared longitudinally witii svstem 

"^""f "^^'^ « P^'^We to mak^^^udgemJnSm 

the health of an educanonal system (Goertz, 1989; Oakes, 1986). "^'^'"^"i 

Su'S^i^^^ '^^^^P^^*^ systems should include data on 

educaaonal inpute, proccf les and outcomes (Fraser. Walberg, Welch, and Ham°e 

ovL^H^lllP^'^'y''' 1989; Walberg. 1983.) /npii/. includf vSs 
cJ^Jn^Ji-cH ^1^ '^''^'^ ^"^'^ i'^"^^'^^ «"ch as resources and teacher 

^i^, ^^^^ as contextual factors over which jurisdictions have no 

^?£?Lr" f nS'-'^'"!"^^^^ socio^nomic siams . Processes are defined 
1987) lis 1^^.",?.^ "'^P^^tf ^'^n^^ ,«> *c oppommities to learn (Fraser et al. 
i,iS„]^i ^^^"^ ^"i""^ vanables as classroom and teaching practices, and 

Snt SiS^^H.'^^T?^^- ^"'"^'^^^ "^^'^ «^"1« achieved inc ucuSg 
smoent achievement, pamapanon rates and attitudes. 

JlSiri?^ identifies a number of characteristics of effective quality indicators 
hdicators shomd : measure a central feature of the system (Office of Education 
Research and Improvement, 1988). provide a benchmik for dcSmiSigproS 
or regression over time, and be understood by all concerned vSS^tiSS^ms 

&'rll?!f Q?^^^ ^"^^^ ^.r«^s2lfS^'a 

o^^Sf^o avnSf goals Of schooling, and limited to a smaU number 

ot measures .o avoid interference with instruction (David. 1986) These svstems 

^S(Da^l98fi?\r?^''i''^^ and district staff in developing the indicator 
JariaWeTm^t' if tJL^ ^.""^^ ""P?^g education, input and process 
vanables must be related to outcomes (Haenel and Katzcnmeyerf 1989; Goertz. 
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The literanire suggests three general methodologies for collection of information 
(Buccino, 1989; Overgaard, 1988): review of documents such as annual reports, 
census data and records; testing which includes provincial assessments, diploma 
examinations, standardized tests and locally developed tests, and surveys including 
opinion questicMinaires, interviews and polls. 



To fulfill the demand for accountability and to demonstrate the effectiveness of the 
school systems, the literature suggests that care must be taken to design and publish 
appropriate reports. Goertz (1989) suggests a number of factors that should be 
addressed in designing these reports: 



• primary purpose 

• primary approach (descriptive/analytical or both) 

• intended audience 

• finequency of publication 

• level of reporting units ( district / school repOTting). 

From the reports on quality indicator system projects, it is evident that effective 
presentation of the data includes a combination of methods; a comprehensive 
printed public report; presentations to community groups by trustees and 
administrators; newspaper, radio and television clips summarizing the results of the 
review; and school newsletters to parents. 



The literature review on inputs and processes that affect student learning indicates 
thr^ categories of findings: smdeni aptitude related variables; instructional process 
variables (including instructional methodology and organization, and school and 
system effectiveness); and context relatedvaxiablcs such as home environment, peer 
group and parenul attitude toward education. 



The literature relative to smdent aptimde suggests that intellectual ability, 
developmental stage and motivation (Fraser et al, 1987) are strong indicators of 
student success. As well, many instructional mediodology and organization 
variables were found to positively influence student achievement including: 
reinforcement and cooperative learning (Fraser et al, 1987); time on task and 
mdividuaiized instruction (Walberg,1983); and wait time (Wise and Okey, 1983). 
Additionally, the school effectiveness literanire idcntiifies a number of otiier faaors 
which positively affect outcomes including: teacher inservice (Sweitzer and 
Anderson, 1983), focus on learning (CHiandler, 1988), varied instructional meUiods 
that are responsive to student learning needs (Chandler, 1988), high expectations 
(National Regional Educational Laboratory, 1985) and a positive working 
environment (Mortimore, 1985). A nu: jber of contextual factors were found to 
correlate positively with snident learning including: the family socio-economic 
stams (Heyneman and Loxley, 1983); peer group attimde toward achievement; 
home environmental factors such as parental suppon of schooling (Fraser et al. 
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ktM^^^'ri^? ^^^^ development and implementation of a quality 
^tf^ T*"" wmbmcd with a review of the research on inputs and pnSS 

^riimoi^ P"^'^" n«^ssaiy infonn^on m de^H 

conceptual fiamcwork for the project which foUows in the next section ^ 

School System Review: Development of the Components 

TJc concepnial framework, predicated on the sources that have prcviouslv been 
cited^rxmdes overaU guidance to the development Sitfie wiSente of the 

^?^^iJeT5f a^irio^"^^ '^^^S indicator? 

Concq)tual Framework 

SSi.m^^i J^?*" ^'^^^ is based on an educational quality indicator 

system consistoig of a set of indicators and related methodoloeTto^Sf rhi 

^SJL?^ A j°f?^^°" ^° ^^^^ ^ "assessing the qasUty of educational 
SSoffQstp '^^"^ '^^^ onLden^out^mes" Sew'S! 

Specifically, an educational quaUty indicator system should include: 

* of provincial goals of schooling and a ranee of 
cognitive affective and behavioral goals; b <>■ «nge or 

* S??r"'^?°" ""^"^^ of indicators, to increase interpretative power 
SfSl iSfn'^n ' "^o^aWenumber of measures for which vLKd 
useiui iniormation can be gathered; 

* ^3^^Li?ni;?i"T "^^^ enduring, easily understood, feasiblv 
n«asurcd and generally accepted as valid statistics; 

* LC\w 3"^°" ^, ^ ^^g*^ Srou-) or a previous result, and 
include input and process variables to assist in interprethig outeomes; 

* dS?" ^ P^^'^S improvement and policy 
SpS?^mpS™°"' interpretation of results , -d development of 
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• a repOTtingsystm that is accurate and timely wthappropi^ 
all administrative units. 



The conceptual framewoik for the school system review process consists of five 
distina but intentlated components: 

1 . a plan fta- involvement of partners in the specification of goals, identification 
of indicators and comparators, intcrjactation of results and the development 
of a follow-up maprovement plan; 

2. a comprehensive statement of die goals of stodent learning including 
OTgnitive, affective and behaviwal goals with a delineation of goals that are 
the primary responsibility of the school and those that are shared with the 
family and community; 

3. a system of indicators that measuijs all of the goals of student learning; 

4. an interpretative framework including standards as well as context, input 
and process variables tiiat affect achievement, and 

5. a plan for the future that identifies strategies and related action plans to 
enhance student performance in areas of strength and to improve 
performance in areas rcq :iring attenticm. 

TTie educational partners wiU be involved in the develqmwnt of all components of 
the school system review process: die specification of goals, identification of 
indicators and standards, interpretation of results and the development of I'n 
improvement plan. Local committees in each jurisdiction,representing a broad 
cross-section of the educational community,will be involved in die developing the 
review process, in sanctioning directions for the project and in communicating the 
project progress to their groups. Additionally, dirca involvement of parents, 
teachers and smdents in development of some components of the project will occur. 



The development of a comprehensive statement of goals of smdent learning by the 
school jurisdictions is a necessary starting point for the system review. Qearly, 
evaluation must be based upon specific results that the jimsdictions wish to achieve 
tahng mto account provincially mandated goals and curricula. Accordingly, goals 
of schooling will be developed utilizing a process that provides die educational 
partners widi the opportunity to meaningfully interpret provincial goals within the 
lo«a context The comprehensive goals satement will include cognitive, affective 
and behavioral goals of student learning )md deUneate which goals are the primary 
responsibUity of die school and which are a shared responsibility between die 
school, family and community. This distinction is necessary to meaningfully 
interpret and understand die outcomes achieved, as well as to assign responsibility 
for improvement plans. 



Once the goals of smdent learning ace established diey will become die focus for die 
development of a system of quality indicators designed to measure die degree to 
which all of die agreed upon goals are being achieved. The system of indicators 
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lify^iHiSl^^ cUrcct and fiiU involvement of parent and staff and 
^i^J^ by the quaUties previously identified in the introduction tote 

^SSt&?op2""^ ^"'^-^ 

of aniculamig system goals and appn)priate indicators or clusters 

of iiufacatOTs, to measure the degree to which established goals are being achievS 
Ae mdK^ system wiU incl^le an intapretatwe ftamew^ includSgSS 

?USh^hS??„?J2:^^^ student IcSem^ 

Through this inteiiJretaQve process, educational outcomes can be fuUy understood 

^wl'Ji'S xelatioflship to the local context - which tendftobe 

r^ly fixed, and mputs and processes, which can be modified to achieve de^ 

Sl^^t'jf S'^'k'^'^'" " established and the student achievement of goals is 
SSSSn^".?-"*^?^"^^ steering colmitt«S 

^.^flSin^T^? ^ • ^ student performance^ 

SfS^ ^ ™P^^ performance on goals requiriniattention A 

flKxiified strategic plannmgpnx:ess (Cook, 1988) WiU includcf » ^ 

• reviewing the strategic policies of district; 

• conq)leting internal and extanal environment assessments; 

• Sr^^ achieved by the distria in relation to the estabUshed 

• identifying critical issues and coiKcrr - 

• establishing specific improvement goals; 

• developing strategies to achieve the goals, and 

• ^^It^ thceducation quality indicator system to ensure that it will 
provide required data to monitor progress and performance. 

Developing the Components 

TJc following section indicates the process of developing the specific components 

Tde^tiS^^.S^^S ^"'^ "^^^^^ °f sSt iSg ^Sid 

Identifying indicators, measures and comparators. The section concludes v^th a 
descnpnon of the mtetpretative framewo* and process. "^nc^uoes wim a 

Comprehensive Statement of Goals: The project steering committee in the Countv 
of Lacombe and the Rocky Mountain Schoil Division met forlw^fiSf dfys to 

^i^««^r^m '^"^'"^ °f smdent learnSig fa devdoping 

Ae ^^ . nfn?S^' ««sidered a number of god statements fern 

the Department of Education mcluding: the goals of schoolini,ihe shared goals of 
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educatton, the goals of elementary and secondary education, the statement of 
desirable personal characteristics of studwits, and essential concepts, skills and 
atntudes of senior high students. 



To assist the committees in dealing with these various statements, the project 
coordinator regrouped all of statements into four categories of goais:cognitivc, 
social, personal and vocational (areas of overlap were noted). In addition the 
committees were provided with guidelines for developing and assessing goal 
statements: * 

• any goal must inchide die intents of provincial goals; 

• local goals can be added to the provincial stattmaits; 

• the goals of schooling as well as die goals of education must be evaluated; 

• each goal must be mutually exclusive; 

• any comprehensive statement of goals must include: cognitive, behavioral 
and affecttve goals; 

• all goals must be measurable. 

The group processes tiiat were organized resulted in gaining consensus on fourteen 
goals m the County of Lacombe and fifteen in Rocky Mountain School Division. 



Following this developmental process the goals were communicated to various 
groups by the steering committee members as well as through Superintendents' 
newsletters. Although no feedback was received at diis point, the goal statements 
were refined m the next stage of die process when indicators and measures where 
identified. Participants at this stage found overlap between goals and concluded 
• ^ measurable. As a result die statement of goals, which is 

included m die Appendix, has been modified and now contains eight goals in each 
school system. 



The significance of this exercise of developing a goal statement is diat these 
junsdicaons now have a statement of goals of smdcnt learning widi a local flavor 
that all educational panners understand and accept as die basis for educational 
programs and die starting point for identifying indicators essential to die review 
process. 



The Process for Identifying Indicators and Measures: The purpose of diis 
development was to identify outcome indicators and existing or required measures 
to assess student achievement on die previously specified goals of student learning. 
Meettngs mvolving parents and teachers in die County of Lacombe and Rocky 
Mountain School Division were held to identify educational quality indicators, 
existmg or required measures and comparators. Workshop sessions were held in 
each school in the tv.-o jurisdictions at which time die project was oudined. 
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indicators and measures were identified, and recommendations for other necessary 
measurement mstruments were considered. u^-c^adiy 

At the meetings after an initial presentation to explain the project, answer questions 
^ individual goals of student learning were then assigned to 
groras of five people to consider, dscuss and niakcrecommcndati^ pSsome 
of toe sessions the groups were organized on the basis of subject interest and began 
their discussions on the goals spcdficaUy related to the subject or group of subjoB 
^u^'J*^ P"^" by starting with the more abstract 

^..^it Tii^ ^ ^ '° ^ '"^j*^ ^ ^» °f ^^^S indicators^" 
measures to dcsoibe them were accumulated as weU as requirements for new 

SSli'S; ^ "^^^^u^^ P"»^^«* to n«t PDup. which 

conadertd the goal cithw at the begmning of their discussions or at the conchision. 
At the end of the group discussion, a general closure session was conducted to tie 
the process together and to encourage on-going discussion and submissions. 

In addition to identifying indicators and measures, tiic meetings succeeded in 
mcreasmg awareness and suppon for the project and the use of multiple indicators 

Stv o/Li:;Stf.f "-S^ T"™ "'^S single indicators to describe the 
quality of education. They also served as a catalyst for participants to think about 
of f for m;complishing. As a reklt Ac statement of goals 

fnnfi^S Suggestions were made about clusteFs of 

indicators and interpretation of results, such as tiic relationship between 

cfS??^^*"" ^l'^^ 8^ at which measures 

ZZ T^^^^ "f^ °! '"^^y instruments. TTic prx)cess also served to 
SJfriln^ ?J7S?f ^•'^ ^t that quality indicators are a reaUty of life at 
this tone and that it w important for them to be involved in tiie identification of 
smtable measures. The opportunity for participants to influence and mold the 
review system so that it is suited the schools of uSc area was considerSi eTSmdy 

After iiiitial identification of the indicators and measures by parents and teachers, a 
tuu review by project personnel was conducted to ensure consistency &om eoal to 
goal and to ensure necessary measures of performance for each goal. These 
commSr«^nH"'';^"'^'jr^" then reviewed by each of the l<S:al steering 
1 '',!fl^°'!^,^ ^ P"ority- The high priorities weiS 

Although an initial set of indicators and measures for snident outcomes had been 
identified, it was rccogmzed that further review and development was necessary to 
ensure tiiat approimate measures were identified and developed. In particular, the 
recent myentory of assessment instniments (Alberta Education, 1990) needed to be 
o^1^.1^? '^^^ whether there were suitable insmmients for^SjSiS 
t^t Jh^A^V^^- /"rth^ore. investigation of possible use of m^sures 
nt^ Educanonal Quahty Indicator projects needed to be considered. 
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The Indicators, Measures and Comparators: Through the workshop sessions and 
the review of the literature, a list of outcome indicators relevant to any system 
review were identified. The indicators were identified in three categories: cognitive, 
affective and behavioral. The cognitive indicators included achievement, program 
participation, completion rates, independent learning and problem solving; the 
affective indicators were student satisfaction, self-concept, self-esteem, values, 
attitudes, inotivation and aspirations; the behavioral indicators were desirable 
characteristics, attendance and locus, community SCTvice, image, health, fimess, 
participation in related activities and post seccmdary registrations. Appropriate 
comparators were also identified for ail indicators. Table 1 gives an example of 
clusters of indicators, tiie measures and comparators for one goal of learning: 
developing knowledge, skills and attitudes in language and cOTamunication. 



The measures to describe student outcome indicators generally fall into three major 
categories: 

• tests which include diploma examinations, provincial achievement tests, 
sta n da r dized tests and locally developed tests; 

• existing documents such as records and reports; and 

• surveys including q)imon questionnaires and participation in activities. 



In general, there ^jpear to be sufficient measures tfiat are readily available in each of 
the three areas. The five exceptions are measures fon speaking and listening; 
wnting skills for grades 8 and 11; commitment to the uso of resources and 
preservation of the environment; fitoess, nutrition and hygiene and healthy 
lifestyles. Additionally, the survey instruments for obtaining opinions from 
snidents, parents, the public, employees and alumni - .^ded to be refined while at 

the same time ensuring consistency with the previc • - adminisiercd instruments 
in the jurisdictions. 



Infonnation will be collected directly through examinations and surveys; from 
school personnel, eg. attendance and enrollment data, discipline data and school 
activities tiiat prranote goals; and other data such as census data will be gathered by 
the project personnel through document review. Following development or 
idcntificaaon of indicators and measures, it will be necessary to fully develop the 
logistics of the collection procedures. At this point in the project, the context, input 
and piocess variables infiuencing student outcomes have not been identified- Once 
more, it will be vital to have full involvement of teachers and parents in this 
proce^ to buUd acceptance for the specific indicators. When these indicators are 
Identified, it will then be possible to describe Uie analysis of the collected 
infonnation essential to the interpretation process which follows. 
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Table 1 

Examples Of Outcomes, Indicators, Measures, Comparators, And Grades 
Measures Comparators 



Grades 



cognitive 



achievement 



student participation 
in regular, enriched and 
remedial programs 



CTBS Listening Skills 

language arts achievement test 
speaking and listening tests 
writing skills competency tests 
English 30 and 33 diploma exams 



docun)ent review 



national norms, past 2 
district and school results 

provincial and district averages 3, 6 & 9 

district results 4 & 7 

district results 8& 11 

provincial, district and 1 2 
past district averages 

district average 3, 6 & 9 



affective 



satisfaction 
completion rate 



attitude survey 
docunnent review 



district and past survey results 3, 6, 9 & 12 
provincial averages Jr, & Sr. High 



2 
o 

§ 

IP 



5 c 
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behavioral amount of leisure reading 

participation in related 
activities 

image * educational 
partner attitudes 

post secondary participation 



library use survey 
student survey 

adult survey 



post secondary reports -docunient 
review 



district average 
district average 

past survey results 

provincial average 



3, 6,9 & 12 
3, 6,9& 12 

N/A 

Post. Sec. 

— 5fH- 
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The Interpretative Framework and Process: The interpretative framework, which is 
proposed, is designed to facilitate an understanding of student outcome indicators 
on the goals of student learning. In each jurisdicticn and in each school the student 
outcomes will be reviewed in relation to the identified comparators as well as 
process, input and context variables to determine whether the outcomes arc 
appropriate. In this evaluative process the variables will also be identified that arc 
ccmtributing to the outcomes. 



An intermetative team, comprising representatives of all educational parmers, will 
be foraied for each goal of student learning in each school jurisdiction. After all of 
the teams have met to interpret the outcomes on spwiific goals, tiie steering 
committee in each jurisdiction will review the results of tiie individual interpretation 
teams to identify overall strengtiis, weaknesses and the process, input and context 
variables contributing to the outcomes. Following tiiese two interpretation 
processes each school wiL interpret its student outcomes tiirough a similar structure 
and process. 



The process of interpretation will consist of five interrelated steps: 

• COT^letionof the tesis and surveys by the interpretation teams members; 

• establishment of acceptable and deshiable levels of performance; 

• determinatira of the relationship of tiie outcomes to die comparators and the 
interrelationship of the indicators for each goal; 

• determination of the process, input and context variables contributing to the 
outcomes; 

• identification of strategies to improve the outcomes. 



The interpretation process for individual goals will commence with tiie team 
members actually doing each test and completing the survey. This activity, carried 
out prior to die team meeting, familiarizes die participants witii the test or survey as 
well as die level of difficulty of die items. Next participants agree on botii an 
acceptable and a desirable level of performance for both die item and its aggregate, 
die outcome. Having die interpreters establish diese levels before diey see the 
acniai results not only deepens dieir understanding of die test or survey but it also 
reduces dicir tendency to rationalize ratha dian interpret die acmal results. 



As a baseline for discussion, die initial interpretative activity in this step will be to 
determine whedier die outcomes arc above or below die identified comparator. This 
wdl be followed by an examination of die interrelationship of die indicators for each 
goal to determine a tentative level of performance on die goal. For example, die 
interpretation of die smdent outeomes for English 30 could be as follows: The 
system results on die English 30 diploma exam were five percent above die 
provincial average and one percent above die past district average. These averages 
were identified as die comparators. While this result would appear to be very 
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S^?c fn.^SiTi*fi'^°^ "^P"^ need to be cx&tnined. On fimhcr examination. 
It was found L«i?t the pamcipation rate for English 30 was two percent above the 

f^^^J^f^'uP^^Z^J""^"*"^' Funhcrmore.areviewofthesmdent 
survey r suits showed that the student attitude and satisfactioa with English was 
very good, and student involvement in rek.ed activities (such as the smdent 
newspq)cr) was high, 

^,Ii5iLP?*"L** inte'Fctation team would be required to make an evaluative 
1**^11^ the student outcomes on a four-point scale. In this instance they 
^.^ P°^^^« ««i of the scale. On the oth« 
,f?^clo not uniformly positive, the evaluation of the outcomes 

nl^J^^^S?"^ ^'^y ^ « ^ «^e^ of the process, input and 

context vanables was completed in the next step. *mu 

™rSL**i^ '*^*Y 9^ relationship of the student outcomes to the 
3 wT.^^ interrelationship of the indicators, die team will determine the 
^ contributing to the positive rcsiCts or identify the variables tiiat are 

^aSS,^« »?^"'' w ^ e'wniining die 

vanables most amenable to change, die instructional processes, to determine die 
^ ST contnbutmg to positive outcomes and diose diat are not In 
addinon, parental activities diat contribute to outcomes wiU be reviewed at diis 

SS!H n '^J!^ "S?"^ ''"^S^" '^on^e^ factors will be reviewed to 

aetcrmmc die efifcct on student outcaues. 

Atthe conclusion of Uiis step, die inteipretation team will be able to identify die 
S?^S?;i!lS"o ^'^.f variables diat are contributing to die smdent outcomes 
for the goal as well as dios-s diat are not Furdiermore, die team wiU be able to 
make a final assessment of student performance on die goal 

tl^Jf'^^ °V^^ ^""'^"^ P^^' ">P"^ and context variables related to 
student outcomes, the team will identify are25 requiring improvements For 

f3M^h.^r^^ diat die prirnary mode oFSstruction in sociSdies is 
centered lecninng and, dierefore, more extensive utilization of student - 
SiS^^nS^T " '"''^ ^ cooperative learning and peer coaching would be 
recommended to improve student performance. These areas su|eested for 

' ^"""^^ ^ "^^^'^^"^ as a part of die system's SveSl 
strategic planmng process. ^ 

Follo^wJlg die interpretation of die outeomes for each goal, die system steering 

SS^v'^i^tS.l- ""^^ °^ *f interpretation piiess to ensui; 

consistency m mterprctation from goal to goal and to identify common strengdis, 
weabiesses and improvement strategies across die goals. The process employed 
in«^!lS?£ * °f bf the in/vidual 

^iSS? H.ff^J^**' ^ '^-^^rpretation. The conclusions of die steering 
rev^ ^ *® substance of die public repon on die systein 
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The final mterprctation activity will be the school level interpretation of student 
outcomes in relation to the system interpretations. A similar interpretation process 
wUi be used at school level as at the system level, that is , teams for each goal and 
an overall review team, both involving representatives of all educational partners. 
The purpose of this activity wiU be to identify individual school strengths, 
weaknesses and improvement strategies. This information will be reported to the 
school's parents and school community. 



Future Developments, Observations and Comments 

The primary activity during the first year of the proje« was to conceptualize and 
develop a fiill school system review process based on a quality indicator system. 
The mtoit was to develop a comprehensive process commencing with delineation of 
goals of student learning, identification and dcvelopnaent of indicators, description 
of coUecnon, analysis and interpretative procedures and development of a strategic 
plan to mamtain strengths and improve on weaknesses. All developmental activities 
were to involve die steering committees in each jurisdiction as well as the 
educational parmers directly for components where enhancing acceptance and 
ownership of the developments was vital 



Developments to be Compleud 



A number of components in the developmental phase still require completion. The 
measures for some student outcome indicators require further development unless 
existmg vahd and rehable instruments can be identified As well, the context, input 
and process mdicators have to be determined by direct consultation with parents and 
educators. Once these developments are completed the collection and analysis 
procedures for measures of all indicators can be finalized. Furthermore, the 
reporting procedures and the follow-up planning process need to be developed. All 
of these activities wiU occur prior to field testing and refinement of the instruments 
in year two. 



Observations and Comments 



Those mvolved with the project believe tha the comprehensive school review 
process based on an mdicaior system has the potential for positively influencing 
student leammg and public support for education. The fact that parents, educators, 
support personnel and trustees have been significantly involved in the 
developinental phase will enhance acceptance and ownership ana increase the 
likelihood Uiat teachmg and learning, policy dumions and parr,ntal ;jid public 
support for education will change positively as a result of the revie v. 
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5. 1 °^ ^^^^ ^"^^ consistent with the Uterature, fimhcr ii^crcase 

fc^^iJ?^ of acceptance and success. Initiating the projea on a sound basis of 

foUowing with identification of clusters of outcome 
mdicators for each goal, identifying the context, input and process variables, 
measuring the outcomes, mterprciing the results and developing a strategic plan fo^ 
K;iTf."^'' f*" as a fair and comprehensive process. In particui^ it should 
neZ^n^ ^f'l'i."^ ^''y^ ^^"^ ^^"^ concerned that judgements about the 
fnHJniS^ -^^^ ^'^""tional system arc often made on the basis of single 
intoors .This review process based on multiple indicators o. the school sy 
process will recdfy that concern. ' 

An additional advantage of the system review process is that it is "incorporated into 
XT^r^Ur!^^ ^ ail organizational effort" (Kaufman, 1988, p.80). In fact, 
• 1^ n" fufldamcntai purpose of school systems, student 
learmng. it mvolvcs aU partners in development and implementiuion, aid it -mil 
result m a strategic plan for improvement The components of that plan v/ill be 
!S?n?JiS^M'°4%'^°^ responsibilities of personnel. This system, therefore, has 
n^lESSl J° the acuons of parent, educators and trustees. That 

potential is enhanced because of the broad acceptance of the project 

^S'?hS'-^-^""^"'^*'^*'^°P^"fe P">j«^ ^ to be cxfixnscd to 
S ic indicator system is manageable from the educator's perspective. 
S l^K^"^"^ a comprehensive set of indicators and meS there 

SJ^helmfnf "??Vk '""^^ """^^^^^ of indicato^is not 

thr^^t^S^S^' ^^^^ suggested that distribution of outcome measures 
Uxroughout the grades wiU assist in making the review process more manageable 
^Sess^s'^L'S^n'f > while participation in the developS 

SSg " become a burden to those fully eiPigM in 

PnTfn'Ti^; ^^°u^ ^"PP°" Pi"oj«^^ in the school jurisdictions. The 

fnS;fv^eT^.^'^'^^ opportunity to mold the nature of tSe 

indicator system. They beheve that once implementjtd, the system will provide 
rv^H?*;??'- ^^''"P^^hensive picture of each jurisdStion's perfor^^cr^nd 
f^Zta '"f°™^"°" n'^cessary to the continual improvement of student 
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Appendix 
County of Lacombe and 
Rocky Mountain School Division 
Goals of Student Learning 

QoalAl Develop knowledge, skills, and positive attitudes in language and 

communication, mathranancs, practical and fine arts, sciences, and social 
studies. 

QaaiZl Develop different modes of inquiry and learning including: 

• skills of finding, comprehending, organizing, analyzing, and applying 
information; 

• skills of learning through technology, 

• skill of studying. 

Qaal2l Develop inteileciuai curiosity and a desire for independent life-long learning. 
fisalA Develop a sense of community responsibility which embraces: 

• an understanding of and appropriate particip-'tion in citizenship at the 
local, national, and international levels; 

• respca for law and autiiority, public and private property, and the 
opinions and rights of otiicrs; 

• ^jprcciation of the hnpcfftancc of traditional and culture. 

QoaL^ Develop a commitm«it to the careful use of natural resources and to the 
preservation of die physical environment 

QoaLk Acquire knowledge and develop skills which contribute to physical weU- 
being. 

QMll Acquiie knowledge, develop skills, attitudes, and habits tiiat contribute to 
eawtional well-being including: 

» achieving a positive self-concept; 

• acquiring a high level of self-discipline and individual responsibility; 

• acquiring an ability to respond to change; 

• developing shon and long-term personal goals. 

QaMx Acquire knowledge and skills, attinides, and habits for individuals to be 
successful and respond to die opportunities and expectations of die world. 

Overall Goal: Achieve successful graduation from senior high school. 
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The purpose of Ais paper is to outline a conceptualization of social competence and 
a methodolosr for tneasuring social con^tence. Son* discussion will be 
provided of difficulties in accomplishing the conceptualization and methodology, 
as well as a discussion of future stages in tfie application of tiie conception and 
methodology to one school jurisdiction. 

Introduction 

Background 

Fon McMunay Public School Distria #2833 is made up of 13 schools - ten 
elementary^unior schools, two high schools and one special programs school. All 
are located witiiin the city. The District's fundamental purposes arc to ensure that 
all 4,700 students achieve levels of knowledge and skill consistent with their 
varying abilities; that they perform at levels equal to or greater than established 
norms; that they develop positive attitudes towards learning and toward tiiemselves 
and others: and that they develop into productiive citizens. 

The Distria has been involved in tiie development of Student Performance 
Indicators and Standards since 1982. The first Annual System Studait Evaluation 
Repon (1986-1987) was presented to Trustees by administration in October 1987. 
The report drew together a variety of infcHmaticHi lepiesenting the edjicarional 
health of students in the system: academic, behaviOTal/emotional/attitudinal, 
physical, and cultural Some of the systems to identify and collect thj infommtion 
took almost four years to put into place. The first report represent d a major cffon 
to provide "measurable" evidence to the Trustees and the public of ihe health of the 
District's educational efforts. 

Although tile defmed outcomes and standards of tiie District's "indicator system" 
represented a solid basis for future development towards being increasingly 
accountable to tiie ratepayer for educational benefits, some indicators and tiieir 
standards were quickly identified as needing revision and refinement. The 
indicators and standards requiring most revision were in tiie area of social skills, 
behaviors Md attimdes. In particular, tiiere was deemed to be a need to shift from 
"negative" indicators to more "positive" or desirable indicators. This smdy was 
designed to address tiiis need for revision. 

Rationale 

First of all schools appear to be becoming more accountable for knowledge, skill 
and attimdes outcomes related to d variety of topical social areas, such as smoking, 
environment, A.I.D.S., suicide prevention and so fonh. More broadly conceived, 
schools appear to be becoming increasingly responsible for development of socially 
competent young people. Thus tiiere is a practical need to understand what 
consrimtes a socially competent individual, how to assess tiie competencies, and 
how to repon on tiie selected competencies. Additionally, all of tiiis needs to be 
done in ways which arc valid and reliable, yet feasible in an operational sense. 
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Secondly, there is a need for fiinher research into the general construct of social 
competence and its assessment The existing theories concerning social 
competence and skiUs appear to lack cohesiveness and consistency. 

Purpose of the Study 

The purpose of the study was to develop a set of social skills, behaviors and 
atamdc indicators and standards which would then be used to assess the quaUty of 
a selected portion of one school jurisdiction's educational program and deUvery. 

To fiilfiU the study purpose, it is necessary to con^lete the foUowing tasks: 

1 . to identify or develop a set of desirable student social skills; 

2 . to identify or develop measurable outcomes and standards 
for the desirable student social skills; 

3 . to establish a methodology for collecting, analyzmg and 
mterpreting the data; 

4 . to identify or develop strategies for teaching the identified 
desirable student social skiUs; 

5 . to identify or develop a means for reporting information and 
findings to users. 

Social Competence 

Systems Conceptualization 

A systems pCTspective of social competence based on a modified CIPP model 
OVorthen and Sanders, 1987, p.78) was used as the conceptual framework to 
discuss the concepts u. tlii: study, and tiieir interrelationships as conceived and 
schemancally presented in Figure 1.1. 

Worthen and Sanders (1987) note that: 

Until we have soM mformarion about the relative effectiveness of 
the numerous evaluation approaches, choices among alternatives will 
remam a matter of tiie evaluator's preference....(since) diere is 
almost no research to guide one's choice, (pp. 148-149) 

They further state that: 

adherence to any one model ra±cr tiian anotiier is largely a 

statement of philosophy or a profession of faith. 
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Social Environment (Context) 


1. Developmental Level 

2. Audience 

3. Situation 





Social Information 
(Social Knowledge, 
Skills and Attitudes; 
Inputs) 




1. Sclf-rclatcd behaviors 

2. Interpersonal 
behaviors 

3. Task-related 
behaviors 


1 



Social Information 
Processing 




1. Encoding 




2. Interpreting 




i 


3. Generating 
Alternatives 




4. Clioosing 




5. EnacUncnt 





Social Responses 
(Products Ouicomts or Outputs) 



Honest Fair/Just 
Kind 

Tolerant Respectful 
Responsible Forgiving 
Loyal 

Committed to Democratic Ideals 
Open-minded Thinks 
Critically 
Mentally & Physically Fit 
Unselfish Appreciative 
Qealive 
Intellectually Curious 
Persevering Prompt 
Neat 

Cooperative Accepting 
Conserving Atten!ivc 
IndusUious 
Pursues Excellence 
Possesses a strong sense of self-worth 

I 



Figure 1. Systems Model of Competent Social Response 



Since no universally accepted ^proach to the conceptualization and evaluation of 
social competence appears to exist, and since no universally accepted approach to 
evaluation appears to exists a decision was made to use a systems perspective as the 
basic organizing framewoik for the conceptualization. The systems perspective 
was conq)atible m± the beliefs of the snidy team and ^pcarcd to provide good 
information for program development and student evaluation related to social 
competence. The concepts which were pertinent to this study included social 
knowledge, skills and attitudes; social information processing; social responses or 
social outcomes; and social context 

Conceptualization of the Construct. 

The review of tiie literature concerning the namre of soci?I skills and social 
competence quickly identified certain common threads which were pertinent to the 
conceptualization that evolved. 

1 . There are specifiable elements/skills. 

2 . There is a need for die elements/skills to be processed in an 
appropriate manner; the processes may be either cognitive or 
affective. 

3 . The appropriateness of behaviors to specific situations is crucial. 

4. There is no agreed-upon generic social skills listing. Also, any 
list needs to be flexible to some extent to the needs of the user. 

5 . Appropriate social responses are affected by the developmental 
level of the respondent, and the audience for the respondent. 

6. Social responsibility can be learned. 

7 . The whole social competence construct is dynamic, synergistic, 
and organic. 

8. Specific social elements and skills are comprised of specific and 
discrete verbcJ and non-verbal behaviors and entail both effective 
and appropriate initiations and responses. 

9 . The skills are interactive by nature. 

1 0. Deficits and excesses in social response can be specified and 
targettcd for intervention. 

1 1 . Instructors attempting to teach social skills or appropriate social 
responses should adhere to the "relevance of behavior rule" 
(Allyon and Azrin, 1968) which states that instructors should 
teach only those behaviors that will continue to be namrally 
reinforced after training. Adopting this viewpoint assists in 
ensuring that social skills selected for instruction will have some 
intrinsic value to the child, some benefits for the child, and be 
valued by others who would likely reward their occurrence. 
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Social Environment ( Context): The construct of social environment evolved to 
include developmental level, audience, and situarion. The "developmental level" 
came to include concepts of age, language de\ ^lopment, physical developmeni, and 
acadenLc conqwtency. For pracdcal pmposcs, they were defined as Divisions I, II, 
in and T\\ as they relate to school systems. "Situation" was further definec?. in an 
effost to simplify th«5 construct and yet provide some forms of recognition and 
consistency to it widiout making it totally unmanageable. Situations were identified 
as old/new, familiar/unfamiliar, and friendly/hostile. "Audience" came to be 
defined as Uiose individuals receiving die responses. The audiences might include 
peers, teachers, parents or 'significant others'. '.Significant otiiers' could include 
older or younger children, or significant odier adults. 

Social Information (Knowledge, Skills and Attitudes): The constract of social 
information that evolved relied heavily on Reschly and Gresham (1981) who 
identified social con^tence as being con:q)osed of two ccHsponents: adaptive 
behavior and social skills. The social skills were further identified as breaking into 
three categories: interpersonal behaviors, self-related behaviras and task-related 
behaviors. Examples of interpers )nal behaviors included accepting autiiority, 
conversation skills, cooperative beiiaviors and peer relationships. Self-related 
behaviors involved expressing feelings, epical behavior, and positive attitude to 
self. Examples of task-related behaviors included attending behavior, completing 
tasks and followi^tr directions. 

Social Ifformation Processing: While it was recognized tiiat tiiere were a series of 
identifiable skiUs, tiiese in turn required some form of processing and are related to 
what Reschley and Gresham (1981) refer to as 'adaptive behavior'. These adaptive 
behaviors include independent functioning skills, physical development, language 
development and academic competencies. These, in mm, were all recognized as 
operating witiiin some form of context, and tiiat context was defined as die 
developmental level of die respondent as well as die audience which die respondent 
was referencing, and die specific siniation. Therefore a model of social competence 
evolved which included a series of social skill components, adaptive behavior 
processes and contexmal variables. This 'iiodel was reviewed by individuals from 
Albeita Education and suggestions were made regarding potential revisions. The 
adaptive processes which haa been identified as academic competencies, language 
development, physical development and independent functioning skills, were cross- 
referenced to odier aspects of the model. Language development and physical 
development were viewed as sub-sets of die contexmal variable of developmental 
level. Academic competaicy, while obviously having a relationship to social skill 
development, was not deemed to be a crucial focus for die purposes of dus study, 
aldiough there is a recognition diat diere is a coirelation. That left one key adaptive 
behavior process identified as "independent fimctioning skills". In an attempt to 
refine the concept of independent functioning skills, a finther review of die literature 
was undertaken to identify and clari^ what independent functioning skills might 
entail. 
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A selective review of literature concerning cognition, decision-making and problem- 
sohong was undertaken. In the process, an appUcation by Perry and Perry (1987) 
of Dodge's Social Information Processing Model of Social Competence was found. 
The model describes cognitive steps thought necessary to children's appropriate and 
competent action in social situations, namely: 

1 . Encoding social cues. 

2. Interpreting behavior. 

3 . Generating alternative responses. 

4 . Choosing a response after evaluating potential consequences 
of altematives. 

5 . Performing the chosen response. 

A fiitiier description by Perry and Perry (1987) notes tiiat encoding of social cues 
mvolves searching for relevant social information before responding. 
Interpretation, involves giving meaning to the cues intended. Tne response search, 
generates vanous possible behavioral responses to the situation at hand» and these 
responccs can vary in quantity as well as quality. The response decision involves 
choosmg a response after evaluating the potential consequences of each possible 
response. And tiie final step is enactment where tiiere is a behavioral performance 
of the chosen response, and children obviously cannot perform successfully the 
response they have selected as best unless they possess tiie motor and self- 
regulatory capacities to cany it out This conceptualization was deemed to more 
completely define tiie concept of tiie "processing" of tiie social knowledge, skills 
and attitudes which were deemed pertinent to tiic study. 

Social Responses: Definition of social response areas is difficult since tiiey need to 
be acceptable as norms of social behavior tiiat are widely accepted across and within 
different groups in society. Therefore, it was decided to adopt tiie social outcomes 
or norms identified as desirable witiiin Alberta and specified by tiie province in M 
School Act and in tiie Guide to Education . 

The 1988 Albena School Act specifies a code of conduct; 

A snident shall conduct himself so as to reasonably comply with tiie 
following code of conduct: 

a. be diligent in pursuing his snidies; 

b . attend school regularly and punctually; 

c. cooperate fully witii everyone autiiorized by tiie 
Board to provide education programs and 
other services; 

d. comply witii tiie rules of tiie school; 

e. account to his teachers for his condua; 

f. respect tiie rights of otiiers. (p. 11). 

AdditionaUy, some desirable personal charaaeristics, oudined by Alberta Education 
m A Guide to Education (1989), are categorized into tiiree areas: 

a. ethical/moral characteristics; 

b. intellectual characteristics; and 

c. socialy^ersonal charaaeristics. 
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The cthical/moral characteristics included respectful, responsible, fair/just, tolerant, 
honest, land, forgiving, ccranritted to democratic ideals and ioyaL The intellectual 
characteristics includai open-minded, thinks critically, intellectually curious, 
creative, pursues excellence and appreciative* The social/personal characteristics 
included cooperative, accepting, conserving, industrious, possesses a strong sense 
of self-worth, persevering, prompt, neat, attentive, unselfish, and mentally and 
physically fit* 

Additionally, John Raven, (1982), indicates that: 

.♦..the vast majority of teachers, pupils and parents want schools to 
foster such qualities as the willingness and tiie ability to take the 
initiative in introducing change into tiicir society, indq)endena, tiie 
ability to make tiieir own observations and leara without instruction, 
tile ability to apply facts and techniques to new problems, to develop 
tiieir characteristics and personality, and to ensure timt they leave 
school intent on being master of tiien- destinies. (p342). 

Raven states that tiiesc opinions arc "correct", as indicated in a variety of research 
efforts between 1961 and 1979, which identifies tiiese factors as the most important 
qualities for our pupils to develop in relation to tiieir futures at work and in society. 
He furtiier notes, however, tiiat despite this agreement about what should be done, 
most secondary school teachers ne^cct these wider goals and concentrate on 
achieving academic goals which probably represent only a sub-set of what tiic 
majority of people wish to have happening in schools. 

Summary of the Construct: For socially appropriate responses, or competent 
social response, students will be required to process social knowledge, skills and 
attinides and respond in ways tiiat are appropriate to a particular social 
environment When students are perceived and received positively by tiie 
respondent, tiien tiiey will be viewed as socially competent. 

Research Design and Methodology 

The process of assessment of social competence appears to be highly complex. In 
summary, tiie review of the literature notes tiiat: 

1 . No single assessment metiiodology is sufficient There appears to be a need 
for multiple metiiods of assessment Michelson et al . (1983) refer to tiiis as 
a comprehensive assessment strategy, and Schwartz and Kaplan (1981) 
would refer to it as triangulation. 

2 . There are major issues of reliability, validity and practicality in social 
competence assessment The validity issues focus on whetiier or not tiie 
child has only a knowledge of a specific behavior, or can also paform tiie 
behavior under appropriate circumstances. Reliability relates to whetiier or 
not tiie varous 'raters* of social behavior are consistent witii one anotiier 
(for example, adults to children, researcher to parent, parent to teacher, and 
so fortii). The issue of practicality notes tiiat tiie assessment strategy must 
have adequate time, trained personnel, resource materials, and 
administrative capabilities in analysis. If tiiese factors do not exist, 
technical problems, inaccurate or incomplete data sets, and overall reduction 
in quality of the evaluation may occur. 
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On the other hand, pnxcdures which are too time consuming, and which 
tan to pnaduceuseabie information wiU result in failure of staff to implement 
me strategy. From the pomt of view of operationalizing the comprehensive 
assessment strategy m a school context, the issue of practicality is 
lundamental to the effective implementation of the total proccs:^. 

3 . No single informant can provide a valid and reliable perspective. Therefore 
ttiere needs to be assessment input from multiple sources. 

4. The purposes of the assessment need to be clearly defined as either 
diagnostic4ntervention, or both. 

5 . The most amventional practical Msessment approaches may be ^orroc -f 
self-repons and behavior checkUsts. Self-rcpons tap into 'aie child's ' 
Knowledge of social skills and provides inf-rrnation rebicd to their 
cogmtive and affective understanding of social skills. However, they may 
not provide an accurate assessment of tl'e child's everyday performance of 
social skiUs. Behavioral checklists by knowledgeable adults can provide 
information related to the acmal and observable social behaviors. 

6 . A potentially useful, but less conventional method includes peer 
^scssment, although this methodology is cumbersome, relatively new to 
behavior assessment, and may have questionable reliability with young 
children. Nevertheless, it does coUect data from the most important member 
cf the child s social environment and wiU reflect an assessment of actual 
social behavior as perceived by peers. 

7 . Peers, teachers and parents constimte the users of the student's social 
competence and, therefore, arc important data sources regarding the 
student s social competence. 

Ill conclusion, no single approach emerges as the best method for social skills 
assessment m all social contexts. Therefore there is a need for the development of 
a comprehensive assessment strategy which will be valid, reUable and practical; 
based on multiple mformation sources and multiple assessment methods; suitable 
to a ymety of developmental levels, audiences and simations; and focus on the 
laennticanon of social skills, processes and outcomes. 

The major methodological techniques employed in educational measurement include 
quesnonnaircs, interviews, observations and document analysis. In an effon to 
ensure a comprehensive assessment approach", an attempt is being made to use all 
methods m some fashion. 

The methods have been cross-referenced to potential respondents as outlined in 

u extended to determine which of the 

methodA-esponden;s were most applicable to the various sub-systems of the Model 
of Competent Soc.al Response (Figure 1). The results are presented in Figure 3 
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Discussion 



Social Competence 

The study team consists of two teachers, two administrators, two counsellors, a 
Supervisor of Student Services and a senior administiatOT, The smdy team initially 
attempted to come up with a simple listing of desirable social skills and r^sed that 
the range of sldUs was substantial Secondly, the team quickly realised that, in 
tenns of selecting the requisite or desired siaUs, a number of other factors came 
into play such as the situations in which skills needed to be used, how the skills 
were processed, and so forth. At this point, the team decided that a 
conceptualization of social competence was necessary prior to being able to identify 
contributing skills and identiiying desired outcomes. 

The study team, from a job point-of-view, had varying backgrounds. Additionally, 
training was varied, and incUided administration, cunkulum, and education 
psychology. Attempts to conceptualize the area of social COTq)Ctencc resulted 
variety of schematic approaches ranging firom matrices to cubes to concentric 
circles. In each instance, tiie individuals proposing theb conceptual framewcnks 
could not understand why otiicr individuals on the tir?"i were having difSculty 
accepting the conception. Eventually it was realised tiiai each team member, torn 
the point-of-view of their personal styles of learning and training, had a 
concepmally different approach to educational issues including the one of social 
competence. The systems frameworic which came to be used was the one model 
which ail study team members could accept 

At this point, the concepmalization became brokra into segmaits. There was a 
consistent acceptance of the skills area, or what has come to be known as the social 
inforaiation sub-system, as being composed of self-related issues, interpersonal- 
related issues and task-related issues. How the information was to be prxx:essed 
lacked agreement until the Dodge conception of social inforaiation processing was 
identified. Thirdly, tiiere was a general acceptance that the definition by the School 
Act and the Guide to Education of socially desirable outcomes would be suitable 
for our needs. Lastiy, team members agreed that all of these factors operated in 
some context that evenmaily came to be defined as developmental level, situation 
and audience. 

The second major difficulty in lerais of conceptualizing social competence had to do 
with framing tiic situational variable. There is no practical method for 
accommodating all of the various simations in which smdents must operas, and yet 
tiie situation has a great deal to do with how children are defined in terms of their 
social competence. Therefore, a fairiy sinq)listic definition of the situational 
variable as old or new, hostiile or friendly, came to be accepted as a step towards 
practical definition for purposes of instrument development 

In summary, the construct of social competence was approached by a variety of 
individuals with a variety of backgrounds who came to realise that the area of smdy 
was much larger and much more complex than they had originally envisioned 
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Secondly, attempting to understand the concept and to define it came to be a very 
ome-consuming and difficult task. However, through the process of grappling 
with the constmct, the team has basically educated itself as to the various 
components and relationships, and has provided for themselves a solid foundation 
for the rcmainmg developmental work concerning the construct. The next stages 
for construct development will be: 

1 . an identification of specific knowledge, skill and animde aspects of the 
social information sub-system. 

2 . refinement of the social response outcomes. 

Methodology 

There was an eariy lealization by the study team that measurement of social 
competence is extienicly difficult and complex. The measurement wouM require 
multiple methodologies and multiple respondents, and that the methodologies and 
respondents would have to relate to the various sub-systems of Ae conceptual 
fiamework* These features would be compounded by the need for any 
methodologies to be valid, reliable, and practical. The study team initially 
considered the development of its own instruments. However it was recognised 
that instrument development, to make it valid and reliable, is an exceptionally time- 
consuming task which was either beyond the capability or the desires of the 
working group. As a result the committee decided to re-examine, on an itcm-by- 
item basis, all the various assessment instruments which had been identified, and 
which fit certain criteria, 

1 . The instruments would focus on smdents in regular classrooms. 

2. There would be a variety of developmental levels, largely defined as 
Divisions I, U, in and IV. 

3 . Preferably, there would be a variety of sub-scales and items which 
addressed the social response areas identifitd, and would have the potential 
for revision to meet the requirements of the team's particular needs. 

^. The instrament would focus on positive social behavior, as opposed to 
negative social behavior, or lack of social behavior. 

5 . Any instruments would also have, hopefully, instrument variations pertinent 
to the various audiences who might be responding such as parents, teachers 
and smdents. 

The search for instruments has been undertaken but is not completed at tiiis time. 
Two instruments which appear to have some strong possibilities arc the 5e//- 
Perception Profile for Children (1985) and the Self Perception Profile for 
Adolescents (1988)which were developed by Susan Harter. These scales appear 
to fit the noted selection criteria. Additonaliy, in conversation with the author, it 
has been determined that new sub-scales can be developed or that some of the 
existing sub-scales can be dropped. In other words, the profile is open to 
developmental work to match more closely the conception for the study. Lastiy, 
the scale has norms established for Alberta, and those norms will shonly be 
reponed as pan of another project of Alberta Education. 
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A second instrument is being considered at the momenL The instrument is the 
Teacher Rating of Social Skills for Children (TROSS-C) developed by Clarke, 
Grcsham and Elliott (1985). The TROSS-C is a checklist fomiat and would be 
suitable, theicfoie, for obsemtions concerning actual behavior The Barter scales 
relate more to student perception of behavior 

Thirdly, a decision .has been made to review some nomination scales which were 
idcnrified in the original literature review. The nomination scales will be used with 
peers in terms of peer assessment of social competence. Therefore, at this stage of 
the study there is serious consi(fcranon being given to quesriOTuaire, checklist and 
nomination scales, in terms of a selection of established instruments. 

Additionally, interview schedules for use by citiicr the teacher and/or counsellors 
are being considered Lastiy, a document review prcx:ess will be considered as pan 
of the assessment strategy. The documents tiiat codd be reviewed would include 
disciplinary reports or anecdotal teacher records, such as a log or journal, which 
would be available in a semistractured foraiat Needless to say, ±c bulk of our 
work over tiie next few nionths will focus on some refinements of die ccmceptuai 
element for botii the overall construct and for meAodology. 

Once the team has identified instruments which appear to have substance, and 
utility, it will be attempting to combine the various assessment metiiodologies and 
respondents pertinent to tfie various sub-systenis, and to develop an integrative 
mechanism for analyzing die data such tiiat teachers will be able to eventually report 
to parents concerning the social competence of tiieir children as it relates to die 
social competence objectives of die school system. Such a reporting mechanism , 
while still needing to be defined, would most likely be represented on some form of 
report card The major premise behind tius forai of reporting is tiiat, if we can 
sufficiently define die social objectives of a school jurisdiction, and if those social 
objectives are consistent witii die objectives of society, and if schools have a 
responsibility for developing diese objectives and outcomes in students, then as a 
school jurisdiction we should have identifiable means of being able to demonstrate 
to society, to parents, and to children, tiiat we have accomplished die task diat has 
been defined for us by die larger community. Obviously it remains to be seen as 
whetiier or not we can do diis in a fashion tiiat is acceptable to tiiat larger society, 
and to parents, and to do so in a fashion tiiat is relatively straightforward and 
practical from die point-of-vicw of the profession. 
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The Alberta community lives with a conviction that man is uniqus and is uniquely 
related to his world. GeneraUy, but not universally, this expresses itself spiritually, 
through the beUef m a Supreme Being (e.g. God)" (Alberta Education, 1990, p.8). 

The Lethbridge Catholic School District operates with the conviction that providing 
excellent academic education be done "... in an environment of love, self-respect 
and respect for others. " ^ 

This research project is responding to convictions such as "...affective education is 
a necessary condition for effecdve education" (Beane, 1986, p.27). 

In spite of these convictions, the importance of affective education is not reflected 
"1,"!,^'^'?^°°^. "Siting system with anything like the same intensity. Indicators 
of affecuve quality are not systematically recorded or reported as indicators of 
success or achievement. 

This paper ouUines the importance of the affective domain as stated in Alberta 
Education documents. It describes the efforts of the Lethbridge project to identify 
attitudes, observe actions, record behaviors and determine growth in order to reflect 
the importance of affective learning in the reporting system. 

A brief background of the Lethbridge Catholic system shows its concern not only 
with intellectuaJ growth and skill mastery, but also with spiritual, moral and emo- 
tional growth. The project fits well with the Prayer of St. Francis. 

Lord, make me an instrument of Your Peace 

Where there is hatred, let me sow love; 

Where there is injury, pardon; 

Where there is doubt, faith; 

Where there is despair, hope; 

Where there is darkness, light; 

Where there is sadness, joy; 

Some of the Uterature which served as a basis for the project is cited. The rationale 
and procedures followed by the Lethbridge project are described. The behaviors 
which are proposed as indicators of affecUve growth are divided into positive 
responses that persons could make to life, to self, to others, to the world and to 
learning. Smce the project is still developing the theoretical framework and identify- 
ing behaviors to r used as indicators, the observations are limited and the implica- 
tions are posed a questions. 

latroduction 

It is clear from the literature and the documents of Alberta Education and the 
Lethbndge Catholic School District that the affective domain is important. This 
syS ^ ^^^^ enabling teachers to express this importance in the reporting 

Purposes 

The School Handbooks for Elementary, Junior High and Senior High Schools in 
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ft^l^^i^^^^^ Education, 1990) set out six specific goals of schooling. In addi- 
tiM^to the competencies, skiUs and knowledge7oals,^the Handb^ks ^so ifst as 

* Imdkf '"^^^"^^^cs, the practical and fine arts, the sciences and the social 

" f'J'nH^rff,'^ ?lu "'^^ contribute to physical, mental and social weU-being. 
' ^nsWp responsibilities, and benefits of active citi- 

' ^orid" f^orif '^"^ ^ "^"^ ^ opportunities and expectations of the 
One purpose i this project is to find indicators which wiU identify these attitudes. 
JhS^'l!? oth^f±n°^' also list the broader goals of education which are to be 

- develop a sense of responsibility. 

- develop a positive self-image. 

•^''IS »!^Snif J?^\^'^ to be Shared with other agencies in society, however, 
...the actions of teachers and the activities that take place in schools contribute in a 
Se n'^^iS,^? S'"'''^^" of attitiides.- (Alberta Education,^^^ p% ?Ss pu^ a 
large portion of the responsibility for these affective goals on the schools The 

Is o^lf' " the success of l5u?atior! A 

second purpose of this project is to assist the schools in their contribution to the 

The Handbooks for Alberta schools state that "...parents and other groups in societv 

&'S^?n nTr?^i^^ growthTpositive attituS'sSs^ 
(AiDerta fcducation, 1990, p. 8) A third purpose of this project is to devise a svstem 
tt' ^^J?' ? ^^g^P^^^h in theTffective domain ha Seen pface 

their efforts at encouraging positive attitiides have been successful. " 

The Handbooks also state that these attitudes are the prerequisites to the deveioo- 
ment of essential personal characteristics, of which 26 are hst^ TWs p^^^^^^ 
be iSiJdteu^ht the development of desirable persoJS cSLtel" 
be recorded through the use of observable actions and products. 

Background 

SJh'aSd^i^/S^'n^ School system and the Public School system 
Svltem hS^ 1 wth ^^Lf '^iT'""^^^ of education in Lethbiidge. The QthoUc 
totL of 152 teSht. ^ ^""^"^^ ^ Hementary schools wiUi a 

SLi • r and 55 support staff to serve their 2700 stiidents. The Catholic 

S in"^.^^'^'^ S"^?"? ^^^^^^ to the full development of^ch 

stident m Uieir care. People m the schools like to think of the schools as place? of 

r^Z^r. IfT. c i ^1 ?f- Affirmation of Students and Staff advocates that ail 
fffi^.t on ^ A?/^ acknowledge the need for affirmation and practice 

affirmation at all times. "Affirmation aims to prevent a person's failures aSd fS- 
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ings from destroying him." (Appendix A) Their aim is the development of a whole- 
some and authentic self image for each student. 



Programs m the Lethbridge Catholic School District are evaluated to determine 
whether they provide an opportunity for students to develop "a maturing Christian 
response as their schooling proceeds." (Appendix B) These responses are in the 
affective domam, m values, attitudes and predispositions. 

The evaluation of student progress in the Lethbridge Catholic School District is a 
cel^ration of leaxTung which profvides "...a sense of hope and success." (Appendix 
C) The teacher develops a trust relationship tvith the students and evaluates them 
...objectively, consistently, fairly and justly with the goal of instilling hope in the 
students and affirming their God-given talwts." (Appendix Q TTie teacher has the 
responsibility to make recommendations wnich will assist the student's self-actuali- 
zation. Whoi Maslow popularized 'self-actualization' in the I950's he showed how 
It was closely related to self-concq)t, and how it affected behavior. The evaluation 
of student progress m the Lethbridge CathoUc system has been designed to encour- 
age the growth of self-image, to assist in self-actualization, and to affect behavior. 

To assist in this process of affective growth. Principals in the Lethbridge Catholic 
School Distnct are commissioned to work with parents and teachers to recognize 
the talents of all children and to promote their development. Principals assist their 
students "...to sec themselves in relation to the Gospel message of love, and to 
develop a healthy self-concept." (Appendix D) 

These policies are the basis for the Mission Statement which declares that: "We 
stnve to instill a responsible attitude toward the world and its people." 

It is from this background that the Lethbridge CathoUc School District has come to 
the decision to research signs of learning in the affective domain. 

Related Literature 

In 1964, Bloom, Krathwohl and Masia noted the lack of a systematic effort to col- 
lect evidenj^ of growth in affective objectives. They suggested that the intent to 
measure affective growth often deteriorates into grading what can explicitly be 
evaluated, or measunng only dramatic or negative developments. They argued that 
attitudes develop slowly and cause problems for teachers who need r>^ults recorded 
tor each reporting penod. They also queried the use of general terms such as 
...develops an interest in reading...' The scope 'of interest' could range from 
recognmng that someone is reading to a passionate devotion toward reading. 

In 1971, Bloom, Hastings and Madaus stated that progress tests and achievement 
examinations which inform students of their mastery of a subject, tend to build 
students' confidence and belief in their own competence. This affects the students' 
level of interest and motivation to learn more. They show that cognitive outcomes 
and affective outcomes are closely related. However, they prefer formative evalua- 
tion and feedback to students on their progress toward affective goals over summa- 
tive grading of affective behavior. 

Bloom, Madaus and Hastings (1981) urged teachers to write clear affective a5 well 
as cognitive objectives. The use of observable actions or products as objectives 
makes it possible to evaluate the achievement of those objectives. Teachers were 
advised to ensure congruence between the stated and the evaluated objectives. They 
were also to ensure that the affective objectives were desirable ones in relation to 
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the philosophy of the school and the needs of the students. 

Rubin (1973) claimed that the curriculum includes affective learning whether or not 
we wish It. He stated that often high achievement comes at the cost of permanent 
insMunty, anxiety and a defective self-concept. He suggested that if teachers deal 
w_»ii the arfect, students emotional weU being improves and so does their efficiency 
or learning. He proposed a balanced curriculum which integrates knowledge feel- 
ings and behavior. 6 » 

Rubin (1974) said that our attitudes make us what we are. Our beUefs or percep- 
Si^^ ?^ attitudes or predisposition to behave in one way. Our perceptions, 
attitudes, behefs and choices are the cornerstones of affective education. Affective 
educafion is about the learner's attitude toward self, toward life, toward school and 
toward purpose. 

Rubin (1982) stated that cognition is a powerful force in the shaping of attitudes, 
which in turn determine emotional responses. He said that the goal of affective 
education is to bnng cognitive judgment on antecedent onditicos and consequent 
behaviors. He believed that the best teachers are concerned with both the student 

: *u ^S*^^^' ^ ^^^^ instructional procedures in both the cognitive 
and the affective domam. ^ 

Bc^e (1983), Beane (1986) and Beane and Upka (1984) said that self perceptions 
influence school achievement but school achievement influences the perccpdon of 
sell. He contends that gradmg systems which accentuate the Dositive. Pive. feedback 
ot acceptance from sigmficant others, and enhance the self as^leamer'are superior to 
computive grading systems which he alleges are antithetical to personal develop- 
ment and an obstacle to social cooperation. 

Glickman (1987) asked if it was good or effective schools we wanted. He noted 
how the unrehearsed responses to thA question were predominately affective and 
dittered from the formal responses of liumerical data. 

The intention of this rese?xch is to find what those unrehearsed responses are in 
regard to expectations of a quaUty education in Lethbridge. The literattire cited, as 
well as a broad range of other literature, serves as a basis for this research project. 



Description of the Lethbridge Project 

Rationale 

,?fh??f ^^ So^ondary Education and the Goals of Hementary Education as listed 
the Handbooks of Alberta Education (1990) aim to assist stiidents in developing 
positive attitudes. *^ ° 

2f Lethbridge CathoUc School District instruct the schools to instill 
responsible attitudes. 

The desire of the schools to report on affective outcomes is reflected in their report- 
ing systems. The Elementary Schools use anecdotal reporting where affecUve 
outcomes are reported through phrases such as, "...works hard.." or "...is a pleas- 
ure to have in class.." ^ 
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Parent-teacher-student ii.terviews arc used in Senior High, Junior High and Elemen- 
tary Schools to report on affective growth. 

The Junior High School reports affective outcomes by using computer generated 
dcscnpiors. Although some dissatisfaction with this method has been voiced, par- 
ents still see affective descnptors as important indicators of success at school. 

The purpose of this project is to assist the schools in their struggles to report affec- 
tive growth by determining specific behaviors which indicate growth; and by devis- 
ing a comprehensive, spcdfic and precise sysxem for the reporting of growth in the 
affective domain. 

Procedures 

This research project began with a survey and a study of the extensive literature on 
the subject of affective education, its sienifijancc, its problems, its connection with 
other aspects of education and its manifestation in observable actions and products 
The literature study covered a wide span of yean of research, as well as a wide 
xope of types of research into affective outcomes. It included a review of literature 
from the Depaitment of Education, the Lethbridge Sriiool District, Alberta Educa- 
tion, and the popular culture which makes affective outcome into marketable 
products. 

The research then turned to the stakeholders of eduearinn in Lethbridge. Personal 
open-ended, yet structured and focused interviews were'con^duct^"^to~nnd out^hat 
the stakeholders of education in Lethbridge saw as important in education. The 
intemcws were designed to discover indicators of success which were in the affec- 
tive domam. The data collected was descriptors of behavior which were seen to be 
evidence of growth and success at school. The open-ended interviews aUowed for a 
wide vanety of responses. These responses were recorded in plain view of the 
respondents who were asked for elaboration or clarification when required. 

Sampling was purposive rather than random. Interviews were conducted with 90 
persons. Those mterviewed were students from grades 1 to 12, parents from each of 
the eight schools m the district, trustees, teachers and administrators from each 
school, clergv, professionals, business persons, post-secondary educators from the 
Umversity of Lethbndge and the Lethbridge Community CoUege, ard professionals 
involved m Correctional work at such places as the Lethbridge Correctional Center 
and Mental Health Services. 

Students were asked in clear and meaningful language to describe the behavior of 
stuoents who do well in school. Adults 'vekc asked to describe signs a quality 
educauon. The results of the interviews were studied to understand what expecta- 
uons of quah^ were held by the stakeholders in education in Lethbridge. The inter- 
view data of descriptors of quaUty were then categorized into ten areas of similari- 
ty. Those ten areas, were analyzed and consolidated into five categories of behav- 
iors and actions which indicated quality. 

The categories of behaviors were separated into those outcomes which- 

- gave purpose to LIFE in a spiritual sense; 

- enhanced SELF esteem; 

- resulted in better interpersonal relationships with OTHERS; 

- showed concern for society and the larger WORLD; 

- were conducive to more efficient LEARNING. 
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clones of affective outcomes were also referred to in Rubin's research (Rubin, 

Analysis of the interview data permitted the formation of four similar behaviors in 
each of these five categones. The 20 indicators were formed into a theoretical 
framework of signs of learmng m the affective domain, and were taken to 17 more 
teachers to ask for their perceptions. TTie insights of the teachers were used to ckri- 
ty and letmc the bst of behaviors and attitudes which the stakeholders of education 
m L£thbndge saw as mdicators of growth in the affective domain. 

Further analysis of the list of behaviors developed from the inter%'iew data revealed 
a commonahty of behaviors within each of the categories. Within the area of behav- 
lors reqK>ndmg to UFE, SELF, OTHERS, the WORLD, and LEARNING thtre 
were mdicators which: 

- were overall assumptions or beliefs; 

- gave hope and purpose to tiieir actions; 

- showed signs of charity, love or concern; 

- were actions based on comraiunent. 

A new and refined draft of behaviors which are seen as indicators of growtii in the 
affotiyc domam, thra mailed to 40 people for vaUdation. The v^idators were 
a^ted if this hst of 20 behaviors were indicators of growth in tiie affective domain: 
If there were additional mdicators; and if tiiere were more categoiies of behaviors. 

Affective Behaviors 

"® proposed as indicators of gro th in the affective domain. 
Growth js demonstrated: 

L In Response to Spiritual Life, when a person: 

I. Expresses beUef in a system of values which distinguishes right from wrong by 
displaying truth, honesty, mtegrity and inner peace. 

w?»d^^?on*Sng^ purpose in life by using prudence and good judgment in 

3. ^empUfies charity by showing love, acceptance, tolerance and reconciliation to 
otners. 

4. Displays selflessness tiu:ough considerate, responsible and tnistwortiiy behavior. 

I I. In Response to Self, when a person: 

1. Exemplifies confidence, self-wortii and trust by willingly taking risks and ac- 
ceptmg error with secunty, ease and good humor. 

responsibility by accepting consequences of actions, cooperating witii 
leaders or assummg leadership roles. , e 

3. Accepts tiiat a wid- - age of emotions such as joy, sorrow, frustration, anger, 
jealousy or fea: are appropruite human behavior by expressing and underst^id- 
ing emotions with mcreasmg nUiturity. 

4. Develops gif^ and talents by showing diligence in work, perseverance in effort 
and pnde ir. accomplis.iments. 

m. In Respor.% to Otiiers, when a person: 

L Shows respect and consideration for autiiority while understanding the legitima- 
Affective Domain 6 S2 Letiibiidge RCSD 



cy of dissent by choosing appropriate manners to agree or disagree. 

2. Shows appreciation for the accomplishments of others by affirming or encourae- 
ingthem. / e. 

3. Accepts and respects the ideas, rights, property and personhood of others by 
using charitable words and a'.4ions. 

4- Q>ntributes to building school community by sharing ideas and working cooper- 

IV. In Response to the ^brld, when a person: 

1. Displays active citizenship by involvement in volunteer endeavors. 

2. Shows concern for the environment by careful and responsible use of natural 
resources. 

3. Shows a concern for social justice and equality through a sense of feir play, 
social action and giving. 

4. Responds to the expectations of the world of work by being present, punctual 
attenftve, prepared and good humored. 

V. In Response to Learning, when a person: 

1. mes pleasure in the search for truth by being open and receptive to learning, 
atnrmmg it and reflecting it with happiness and enthusiasm. 

2. Displays self-direction and self-motivation by initiating activities, extending 
knowledge, exploring possibilities or creating. 

3. Displays an open and inquisitive mind by accepting challenges, considering 
change, and attempting various problem-solving techniques. 

4. Displays intellectual curiosity thrcugh the use of critical thinking and informed 
questioning. 

pese indicators are m the process of being made more precise, specific, and usable 
by Identifying actions and behaviors which are appropriate as the child matures. 
The behaviors are being identified in consultation with teachers. From this list of 
specific behaviors, it will be possible to devise a system for reporting affective 
growth. 

Discussion 

Observanons 

1. There is intense interest in the affective i^omain. It is evident in the vast scope of 
res^h, debate, and literature on the affective domain and its pervading influ- 
ence m other aspects of learning. 

2. Ii'-terest in the importance of self-image and attitude to success in life comes 
through m the message of the popular culture, which makes the building of self- 
esteem into a marketable product. 

3. The schools in the Lethbridge Catholic Schooi District show interest in the 
affective domain by continually seeking ways of reporting affective growth to 
students and parents. r =. c. 

4. The elementary students of Lethbridge Catholic Schools who were asked to 
descnpe successful students in school used phrases such as 'they try their best' 
and 'they smile and like school' . 
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^* Si!? -^f**®®^ who were asked for evidence of success at school often 

cited interpersonal relationships and behavior as well as attitude toward school. 

^* S?f!L"^^T^' mentioned self-motivation, self-confidence and 

self-esteem as bemg qualities which indicate success at school. 

^* 2;5^^^';'n"nity members cited open-mindedness, intellectual curiosity, values 
and ?ttitude5 as mdicators of quality in education. 

^* !lcl!l?H-^iuy^®r°''^fy^^ importance of the affective domain, there is no 
systematic method of recordmg and reporting affective growth. 

Impliamons 

Thejesearch into signs of learning in the affective domain has raised many ques- 

SJiiS^lish~'"^ '^^"^ — 

2. What instruments can be used to rqwrt on the affective domain? 

^' inservia will be necessary so that teachers wiH be able to adopt a 

system of reporting affective growth. ^ 

4. What manifestations of the indicators are ^ropriate as the child matures? 

Ths enthusiasm with which the project has been received gives hope that the 
^0^^ mstrumentation will be accepted as a means of reporting aff^tive 
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Appendix A 

AFFIRMATION OF STUDENTS AND STAFF 

Lethbridge CathoUc Scl.ool Districv Policy adopted June 13, 1984 

1 



L^!f£r?f Soard of Trustees draws attenion to 

^' dH^;L^^nf,S^ ^^"'^ sacrament of Recon- 

^f^ P^^^l'f^ weaknesses, and in absolution declares people well 
again. P^i^ as a protest against emphasis on the negative aspects of behavior 
^me Catholic writers claim the faiiur? to affirm as "^greS, " ^one 
can affirm and everyone nods affirmation. Most people doS b«t SSey 

fS^^Sc^'K^^of ^^^^^^^ ^ P^"^^ ^" acknowledges the 

^* ^°oJ/our ""1? 1^^^^' ^"^ci^'n on how we 

^h^ln? f S?^' ^-^ members of the School District acknowledge the need 
for the act of affirmauon of each person in the routine performanci ofhis dS 

4. With respect to the student, the school aims to have each one graduate with a 

our scht>pis know Uieir abilities and have joy in them; they know something of 
their hmitations and can appraise their significance. ^meuiing or 

^' Sff ^tf discu^ion of the practice of affirmation of students and 

statt with the entire school staff annually 
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Appendix B 

EVALUATION OF PROGRAMS 

Lethbridge Catholic School District Policy adopted June 12, 1985 

The School Board may require evaluation of a program within its jurisdiction for 
whatever reason it considers appropriate and at the times it considers most useful. 



Background 



From tunc to time, the School Board, its administration or teaching staff may dis- 
^ a n^ to review a particular program in the District. In that event, the School 
Board calls for an evaluation of the program. The School Board has an abiding 
concern to ensure that tlie students develop a maturing Christian response as their 
schooling proceeds, and wants to ensure that the program provides that maturine 
among whatever other program specific objectives belong to it. 

Guidelines 

1. Pro-am means a set of indq)endent activities and services designed to achieve 
specific orgamzauonal goals, poUcies and objectives. Examples of programs 
are: Mathematics 30; Vocational Education, Counselling, transportation and 
school modernization plan. 

2. The evaluation assesses the effectiveness of the activities in the program against 
Its gpals, policies and objectives as specified in the Program of Studies and 
cumculum Gmde. The evaluation comments upon the appropriateness of the 
goals, policies and objectives. Where appropriate, evaluation will determine if 
the program affords the students with a developing understanding of the Gospel 
Message as it applies in the program. 

3. The evaluation describes the program, identifies the number of students served, 
makes commendations and recommendations and determines the costs. 

4. The evaluation group may consist of personnel from th5 School District staff 
along with consultative personnel from Uthbridge Regional Office of Education 
or other school jurisdictions as deemed desirable. 
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Appendix C 

EVALUATION OF STUDENT PROGRESS - THE CELEBRATION OF LEARN- 

Lethbridge Catholic School D istrict PoUcy revised June 12, 1986 

SS!^f^«T'.^^"l?^^°?' reconciliation, and renewal direct and guide the proc- 
oJ?L°iSy^"^^.°!! °^ P"»8^ess and the celebration of learning -m Ss 

PH Sf'''" -5 *f ^^bridge Catholic School District No. 9 ShSfe of Xlbem 
Education provide the basis for these processes. 



Background 



S5^)l ilthn^^^^^^ ^ ^'^^'^^ °^ *e best attention of the 

^Sl^ Pj?"^ ^ P"^^g ^"i^ attention during the 

iS^i™"' ^^^^ recognition of the love parents have for their 
l^c^'^!S^Jl^'^''^r^'^:^'^!^ '^^^ with a sense of hope and 
Sr? if; h^^** ?J ^""'^^u' °J Lethbridge Catholic School District No. 9 sup- 
^mtiS "^^'^ °^ strengthening the students' confidence in their 

Guideline 

^' SristiSS^iliSi'^ ""^"^"^ ^ development of each child's 

^' t^^Tn^^ P.?"^ rcsponsibiHty for the process of evaluation of student 
progress lies with the classroom teacher who develops a relationshio of trust 
with the student and parents. TTie principal suppUes l^e^Wp ^ 

^' I?cf« progress, the classroom teacher does so objectively con- 

4. In the classroom setting the teacher has the responsibiUty and authority to make 
recommendations w' ach seek th., development of the student's Llf aS^S 

5. The classroom teacher communicates clearly, precisely and in confidence such 
recommendations to other teachers, the princf^, parents and smdenis 

6. In the secondary school, the teacher informs the students and parents of the 

inll' fir^"^^°" P"^^- ^ ^^^"^^"^ «^hools, the cS^t^che? 
will mform the parents, upon request, of the basis of the evaluation procesT 

SfhSfilf^.^** t-^°^^ evaluation of student progress as a means of 

celebratmguie achievement of the students. «ia ^ .nc<ui» oi 
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Appendix D 

ROLE OF THE PRINCIPAL 

Lethbridge Catholic School District Policy adopted March 28, 1984 



1. The Principal leads the school: whether he acts or not, ne determines the school 
as no other person does. 

1.1 In a Coolie School the principal creates a climate for the spiritual develop- 
ment of the staff and students; he appliw Gospel values to the management of 
the school, and so aims to assist his students to see themselves in relation to 
the Gospel message of love, and to develop a healthy self-concq)t. 

1.2 The principal serves as instructional leader of the school, he understands and 
accepts the social role of the school. He has a commitment to personal growth 
m education: he leams and studies. 

1 .3 The principal administers the school. He seeks to maintain a collegial rela- 
tionship with other teachers on his staff. He works with other members of the 
administrative staff on the implementation of School Board policies. He 
manages the resources of the school and serves the staff and students. Ke 
provides for the safety and comfort of staff and students. 

1.4 The principal accepts the parents as the primary educators of the children: 
they have the chief responsibility for their children's education and a great 
influence on it. 



2. The Responsibilities of the Role 

2. 1 The principal workc together with the staff, parish priests and religion con- 
sultant to influence the school with the Gospel message. He arranges for 
celebrations acknowledging the liturgical calendar and significant events in the 
Uves of his students and staff. He seeks to develop a supportive environment 
^9 .^.".f.^^.^^*^ develops as a whole person. He understands the meaning of 
visiDihty m the Catholic Community. 

2.2 The principal explains; the school to interested persons. He accounts for the 
achievement of the students. He helps teachers interpret the achievement of 
children. He establishes a climate of openness in the school and welcomes 
visitors to the school. He knows and makes use of the resources and services 
of the community. He works with his staff in the development of an appropri- 
ate array of co-curricular activities. 

2.3 Tni principal supervises instruction, consults with teachers and other 
responsible personnel to ensure the implementation of the curriculum as 
required by law, regulation and School Boord decision. He assists teachers to 
employ new methods and materials. He encourages teachers to invent. The 
pnnci^ identifies in-service needs of staff and arranges ways of meeting 
them. He participates in the selection of staff. He assigns teachers to subject 
areas of greatest teaching effectiveness. 

2.4 pe principal seeks to recognize the talents in each child and promotes their 
development. The principal work? with parents and teachers in identifying and 
developing these talents; he readily consults with them on the needs and 
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opportunities for growth and development of the student. He attempts to match 
cmidren and teachers based upon learning and teaching styles. 

2.5 The principal enjoys and respects the coUcgial relationship he has with other 
administrators and members of the Principals' Association. 

2.6 The principal inspects the school plant and school grounds regularly and 
recommends needed changes. He provides routine supervision of the plant and 
taciuties. He ensures the procurement of materials and supplies for the 
school, directs the preparation of the annual budget, and looks to the ordering 
of those same items from suppliers. ^ 
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